It's possible to max sword damage and still have very high on speed - use trinkets from pvp tokens. There's a maximum damage bonus - handily labelled as "Maximum!" in-game - you reach it with 6 levels of damage bonuses. Typically you'll see Low and Medium on gear - Low is one level, Medium is two levels. 3 Medium bonuses stack to Maximum. For swords, you have a couple options to get there, mainly because there are 5 slots you can have a Medium damage bonus in. Skolver armor and helm both have Medium damage, so does the Barbarous Thorn Shield, and there's Elite Slash Modules from pvp rewards. Pick any three pieces (and you can have two modules, they stack) and you get max sword damage bonus - and you're left with two slots for anything else, for example keeping the speed bonus from Vog.

The speed vs damage argument is fairly simple - speed allows you to attack slightly more often, and block slightly faster between attacks. Stack up a lot and the lagtime for being able to shield with 2 hit swords becomes similar to that of 3 hit swords without a speed bonus. Damage, on the other hand, simply increases your raw output - if it doesn't increase your damage enough to make you kill things faster, it's useless; on the other hand, it could be the difference between 3 and 4 hits per kill, which might be faster than the equivalent speed bonus when it comes to the amount of time between kills.

The key issue with damage bonuses is if they aren't reducing the amount of hits it takes to kill something, they aren't doing anything at all. But the only way to test whether increasing your damage bonus will do that for you is to get the bonus, unless we ever find a way to determine monster hp (Though a max damage bonus is about 33%, so it reduces hits per kill by one per four at the least). Speed bonuses, on the other hand, will always help you shield faster, and if you were to attack constantly without stopping, they'd always reduce the time it takes to kill something. The only issue with it is getting used to the change, because you won't be as efficient until you learn the new timing.

Then there's charge time reduction. Unless OOO update the Troika line's charge attack, worthless for those, but some swords like DA and Leviathan benefit a lot from it. Charge time reduction is actually a little more efficient than damage bonuses - it scales at somewhere around 10% per point, where damage bonuses and speed bonuses are about 6% (if the bomb charge tables scale for all weapons). I imagine that's because it only applies to half a weapon's abilities rather than all of them. A lot of swords and guns can live without charge time reduction, but you do get two points of it for free when a weapon's at level 10, so if you choose to get it, you only need four more points. You can only get those from trinkets and weapon UVs, though.

Maximum is as high as it goes.

For example...
Cold Iron Vanquisher has a high vs undead built into it. Used with no other enhancements to damage, it will do significantly more to undead than anything else.

Now, add Skolver armor and Helm, and a barbarous Thorn Shield (all of which add medium damage...medium > very high > maximum to all swords). Maximum + high = Maximum. My Cold Iron Vanquisher does the same damage to everything on a tier, regardless of type. This goes for Jellies on the Royal Jelly levels...If you equip a weapon like a nightblade to fight Jellies, it actually won't do more damage, as the maximum cap was already hit.

The same applies to speed and Vog. Once you hit Maximum, say with a quickstrike Buckler + Vog, everything is going to go as fast as that particular sized weapon can swing.

me? I would try to get one piece of Skolver and one piece of Vog. You will get a medium speed increase, a medium damage increase, and whichever shield you pick will either add damage or speed (briar shields/quickstrike) I have found this combination to be really nice when dealing with the large weapons like Troikas and Sealed Swords, as you will still have a dominating pile of damage, but be able to get your swings in fast enough to knock your opponent away with knockback.

I wish there was a CTR armor for swords. Snarbolax armor should do that. Either CTR or be like Mad Bomber and have both speed and power at the loss of status protection.

I'm a fan of dark armor/light helm (I wear a white gunslinger hat as costume and ash tail coat. I think it looks pretty sweet)

So skolver helm/vog armor for me.

Still, don't forget the game has costume slots. Get whatever combination you feel like, and if it doesn't look great, get a low level item or set for looks.