There should be group options to allow certain guild ranks to join (in addition to friends / everyone) by setting a minimum rank or setting them individually. This would allow more people to adventure with an 'open' group without having to worry about p.e. recruits popping in accidentally.