Yeah, sounds like you are on a good track for reducing fleetballing. The key to getting gamers to not fleetball, is, as with any strategy in video games, is to make it more profitable to not fleetball than to fleetball. In vanilla sins and its subsequent expansions, fleetballing was usually incentivized. A lot of that, I think, has to do with the pace of the battles in Sins. Battles between two well rounded, well supported, well micro'd fleets can last for quite some time, and taking planets takes quite some time unless you have a massive fleet, so your best bet to do any real damage is to clump as many ships together as you can and prioritize key planets, and in battle, key ships. And of course this was only compounded in expansions which introduced ultra-heavy defensive structures in the form of starbases.
I think the devs kind of tried to reverse this with Titans, but Titans end up becoming so powerful so quickly that your best bet still ends up being to mass most of your fleet around your Titan in case you run into the enemy Titan, which your fleet might not be able to stand against depending on how many levels it has gained. (unless you're facing an advent titan anyway, in which case you just bomber spam and run away before it vaporizes your ships with a brilliant cleansing light/ converts all of them into a doom fleet against you) In EaW, a few ships can, with proper micromanagement ( and improper micromanagement on their opponent's side), cripple a much larger force with good timing and positioning. This is certainly possible in Sins, although it's almost always just due to mass AoE spamming, which isn't something that exists in EaW.
I think that deteriorating performance as a ship is damaged could strike a nice balance between the two, since it would encourage retreating damaged ships if they are not absolutely vital. In combination with the fleet commanders, this might be enough to encourage smaller, more independent units working to achieve the same goal, rather than one huge megaforce just steamrolling everything.