Background depth image
During a calibration step, allow a background depth image to be generated using maximal temporal smoothing, so that it can act as a maximum scene depth and help to anchor walls etc. During the calibration all foreground objects are removed and the scene is captured. The process can be paused and additional texture such as posters can be added or moved around the walls etc so fill in bad areas. The depth map (without colour image) is then saved to 16bit png and used during normal operation to clamp depths and to act as an anchor when filling / smoothing.
It could also be manually edited or generated by an alternative method.
Edited by Nicolas Pope