Also consider a confidence for a given depth that looks at normal (is if facing the camera or not) and depth from camera since larger depths have lower precision. This might allow cameras more directly facing or closer to the object being viewed to have a greater say on the depth value at that point.

Hopefully this isn't all too time consuming.

The "is it facing camera" test is a duplicate of that done in reprojection, perhaps something can be shared.

New renderer will entirely replace this