I've been thinking about this recently. Before the Arsenal Station update, I always carried at least 1 neutral damage weapon around. I was always afraid of situations like getting stuck on a Graveyard with two shadow weapons, so I packed a calibur all the time. Now that I can switch weapons around at every level, I can still bring two shadow weapons into a Graveyard and switch them out for elemental weapons instead. But since neutral damage weapons can simply kick everything's ass, I want some opinions from more experienced players.
iron slug vs callahan
biohazard vs neutralizer
supernova vs polaris
Neutral damage should have no place in the arsenal of anyone who is considering different 5* items.
Even if you were to carry two elemental and two shadow weapons to any given level, you will still be doing neutral damage at LEAST to every enemy you encounter, with critical damage versus 4/6 of those enemies.
Reading into your post, it sounds like you only have two weapon slots. If this is the case, why don't you buy more? If you find weapon and trinket slots on the trade market you can usually get them for around 200 (weapon) and 100 (trinket) crystal energy each. For 600 CE, you could have a complete character loadout of four weapons and two trinkets. There is really no reason to not have them.
For example, in your three considered cases, if you had four weapon slots, you could go with a Callahan, Biohazard and Polaris, plus a sword, like Leviathan or Divine Avenger. This way you could be doing critical damage against every enemy you encounter with guns, as well as having a backup sword for the odd time that you get cornered and need to fight your way out.
If the question is, however, 'Is normal damage worth it if I only have two weapon slots?', my answer will have to change slightly. Specialized damage is still superior, but there are cases in which normal damage could be useful, unlike if you had more than two weapon slots, in which case you should (almost) never use it. It would be useful in the case where you want to only carry one sword, one gun, or one bomb. Carrying one gun which is pure shadow damage is useless if you need to attack fiends from long range, and thus you will have to switch to your sword, which may not be desirable. However, if you intend to carry two swords, two bombs or two guns, then you should pick two specialized damage types, and thus you will never do less than neutral damage.
Hope this helps!