Specular reflection removal in the context of SFM techniques for Architectural Heritage 3D models construction

In the field of Architectural Heritage (AH) 3D model construction and visualisation using low-cost technologies structure-from-motion (SFM) techniques recently be-came a key technology ensuring ease of use and efficient results even for non-professional. Significant progress has been achieved towards point-based SFM tech-niques, primarily in the area of efficient algorithms for scalable image matching, and large-scale bundle adjustment, which is a core component of all SFM approaches. As a result, it is nowadays possible to easily reconstruct large scenes from image sequences. The topic of urban 3D reconstruction and architectural modeling from images, in particular, has received considerable attention in the last decade. Our re-search group proposed last year in this same conference the development of tech-niques to integrate in the SFM pipeline for AH 3D model construction accurate col-our capture, reproduction and visualization using web-based real-time rendering techniques. An unsolved issue by the method explained in the 2013 paper is one of the effects of specular reflections on both dense point clouds generation, and faithful colour re-production. These effects appear in urban environment in areas with polished floors (i.e. marble), or in presence of windows or shop fronts, and determine scattering ef-fects in the generated points and values of diffuse reflectance incorrect. In this paper we present a new technique to remove highlight effect from the SFM results. Our solution provides a new reconstruction pipeline completely integrated in the SFM pipeline re-using existing data to arrange new results. The process of acquisition of the images to get the finished 3D model is therefore unique and the process for acquiring and visualizing the correct perceived color is fully integrated with the process of shape capture. Overall, the method does not require specific technical knowledge and is therefore relatively easy to use and it can be used repeatedly over many situations of the city. Our processing is semi-automatic and keeps the use of the many photographs used for the 3D textured model construction of our traditional SFM pipeline. Main steps of our processing are: a. image denoising; b. image color to gray conversion; c. image highlight removal. This last is the key step of the pipeline and presents main novelty. The proposed method is a high-level image-processing algorithm. As such, it uses several lower-level methods for its building blocks. We consider these methods as black boxes, and we explain below their input, output and purpose. We use SIFT as a method for extracting keypoints from an image and assigning robust descriptors to them. These descriptors can be matched to those of another image in order to produce a list of pairs of points. This list of matching points can be used to estimate the parameters of a geometric transformation between the two images. Typically, a robust estimation method such as RANSAC is used, which automatically rejects wrong matches before computing the model. For our purposes, we use this combination of SIFT and RANSAC to register a pair of images by a homography. Further steps concern histogram equalization, Poisson editing and the final step is a sharpening to enhance the slightly blurry images obtained after the whole pipeline. We demonstrated the efficiency of our method using case study of our work on Bologna’s arcade system.