This has been suggested before, but here's what I'm thinking about the idea.
-----
It would be a building on the far right side of the Garrison, either replacing or being right next to the alchemy machine.
-----
It would function somewhat like the guild hall building, except players wouldn't be able to go in without an invite from the owner.
-----
The player could customize it similarly to a guild hall - After activating edit mode (from... something), they can place or rotate furniture, install rooms, etc.