Augmented Reality meets Deep Learning

Hassan Abu Abu, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger and Carsten Rother

Abstract

The success of deep learning in computer vision is based on the availability of large annotated datasets. To lower the need for hand labeled images, virtually rendered 3D worlds have recently gained popularity. Unfortunately, creating realistic 3D content is challenging on its own and requires significant human effort. In this work, we propose an alternative paradigm which combines real and synthetic data for learning semantic instance segmentation models. Exploiting the fact that not all aspects of the scene are equally important for this task, we propose to augment real-world imagery with virtual objects of the target category. Capturing real-world images at large scale is easy and cheap, and directly provides real background appearances without the need for creating complex 3D models of the environment. We present an efficient procedure to augment these images with virtual objects. This allows us to create realistic composite images which exhibit both realistic background appearance as well as a large number of complex object arrangements. In contrast to modeling complete 3D environments, our data augmentation approach requires only a few user interactions in combination with 3D shapes of the target object category. We demonstrate the utility of the proposed approach for training a state-of-the-art high-capacity deep model for semantic instance segmentation. In particular, we consider the task of segmenting car instances on the KITTI dataset which we have annotated with pixel-accurate ground truth.

Session

Posters

Files

Paper (PDF)

DOI

10.5244/C.31.81
https://dx.doi.org/10.5244/C.31.81

Citation

Bibtex

            @inproceedings{BMVC2017_81,
                title={Augmented Reality meets Deep Learning},
                author={Hassan Abu Abu, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger and Carsten Rother},
                year={2017},
                month={September},
                pages={81.1-81.12},
                articleno={81},
                numpages={12},
                booktitle={Proceedings of the British Machine Vision Conference (BMVC)},
                publisher={BMVA Press},
                editor={Tae-Kyun Kim, Stefanos Zafeiriou, Gabriel Brostow and Krystian Mikolajczyk},
                doi={10.5244/C.31.81},
                isbn={1-901725-60-X},
                url={https://dx.doi.org/10.5244/C.31.81}
            }