Real-time enhancement of sparse 3D maps using a parallel segmentation scheme based on superpixels

In this work, we focus on the problem of feature-based 3D mapping of environments with large textureless regions, which generates sparse 3D maps that may not represent well the mapped scene. To deal with this problem, based on our previous work (Cruz Martinez et al. in 2016 IEEE international symposium on mixed and augmented reality (ISMAR) adjunct proceedings, IEEE, 2016), we propose to enhance sparse 3D maps by using a superpixel-based segmentation with the aim of generating denser 3D maps of the scene in real time. Superpixels are middle-level features, which represent similar regions in an image, which can be connected in order to segment textureless areas. We propose a graphics processor unit architecture for (1) superpixel extraction considering chromatic and depth information, (2) superpixel-based segmentation, generation of connectivity matrix to compute the connected components algorithm and (3) mapping of segmented regions to 3D points. We use the ORB-SLAM system (Mur-Artal et al. in IEEE Trans Robot 31(5):1147–1163, 2015) to generate a sparse 3D map and to project the textureless segments onto it at 27 frames per second. We assessed our approach in terms of segmentation and map quality. Regarding the latter, covered area by the generated map, depth accuracy, and computational performance are reported.