I had the same thought when I went that way in 2009. The 4 mile portage is the only way to go, the portage directly to Mud is grown over. With that said, the 4 mile is not bad. At that time, I ran across an area where the portage was flooded (see below). I pulled through it and kept going to the portage over to Mud, but on the way back I paddled right up the 4 Mile. The portage from the 4 Mile to Mud is pretty narrow and rocky and the portage from Mud to Ella Hall is relatively easy but not used much and overgrown a bit. Ella Hall is definitely worth seeing, Mud not so much.
Spot where you can get to Mud directly from the 4 Mile.
Ella Hall.