It happened to me twice so far, its a quite "annoying" bug. My group settings are the following:
-Blocked
-Not allowing friends
-Not allowing guildmates

Yet with these options, I sometimes have friends that still manage to join my party, and I have no idea why. I double checked I put the right options, and I'm pretty sure I did well.