You cannot prevent players from ragequitting because they've been outclassed by their opponents; what you're describing is not a balance issue, it's a player behaviour issue, and that's rather more difficult to regulate.

Explain how your "one team against waves of enemies" idea would work. Are you talking an all-player squad versus hordes of AI enemies? Or a L4D Versus-style all-player squad versus player-controlled enemies? I'd be interested to see how the latter could work, but the former is, um, kind of how the main game works already.

I play a lot of Blast Network, and in that you often see one or two players riding roughshod over the rest of a group, and you do get the odd ragequit. But, except in the unfortunate cases where people are clearly having their teeth kicked in by lag issues, that's because the winning players are better at the game. It's not a balance issue as such, it's more to do with the relative absence of matchmaking mechanics.

So, perhaps that's how to prevent or at least mitigate ragequits - have player rankings be consistent, and give people the option of playing ranked matches. Lockdown has a rudimentary version of this in the form of its Tiers, but they're purely gear-based; any actual win/loss, kills/deaths stats are disregarded, and that's the closest you can really get to a skill-measuring algorithm in a game like this. So, perhaps allow players to compete in skill-matched games, where they're pitched against opponents with comparable rankings, perhaps for a greater reward to reflect the increased difficulty.