Depth only correspondence
Allow the initial reconstruction correspondence to try depth only to align surfaces. Subsequent searches can use colour. This is an experiment.
At the same time attempt to use the weights channel to adjust depth weightings, perhaps removing depths whose weights are too much lower than the other camera but smoothing similar ones.
Edited by Nicolas Pope