Best way to deal with going downstairs/upstairs in levels?

Would it just make sense to have floors like in a dungeon be stored in their own game object and just set them inactive and the new current floor to active and leave the character where he is? Or would it be better to have the area some far off location in the world? You can also load scenes but that always feels so clunky and slow, even if loaded in the background.