Colour gradient pre smooth
Try another smoothing approach based upon colour gradient as smoothing weight. Repeat iteratively with a tiny kernel size (3x3). Also consider and recalculate normals at each step to use for estimating new point position (an MLS or APSS based smooth over a small window).
Note: This operation could be done on the vision nodes and not in reconstruct. How does it compare with bilateral filter etc, since (if my data is using the filter), that filter still produces noisy results.
Edited by Nicolas Pope