Increased attack speed makes sword users "refresh" faster between combos, but it does not let gunslingers' reload any faster.
This is broken.
Swords already deal more damage than guns. By allowing the offensive bonuses to be more effective for sword users than gunners you're widening that gap well beyond the point of balance. IAS needs to affect reload time, particularly since for some guns that's the entirety of their attack cycle. Some guns literally gain nothing from IAS courtesy of this problem.
Swords put you in the line of fire. Guns keep you far away and safe.
Risk:Reward.