Eff those guys. Eff them hard.
I don't mind them being unkillable by conventional means. I don't necessarily mind them making me rush through a level and miss the loot I came for. But I want to KILL THEM DED and TAKE THEIR STUPID LOOT for making me sweat. And then I want to explore the rest of the level.
So, my suggestion is to have holy water seeded throughout graveyards that instakills phantoms. The party rushes to find it, obliterates the phantoms, and continues playing normally.
Lately I have been trusting my party with the crown/heat gathering, and I just aggro all the phantoms and have 10 minute (freezing) dance battle with them.
Yay for teamplay. I doubt the devs intended the levels to be played like this though.