Play and Learn: Using Video Games to Train Computer Vision Models

Alireza Shafaei, James Little and Mark Schmidt

Abstract

Video games are a compelling source of annotated data as they can readily provide fine-grained groundtruth for diverse tasks. However, it is not clear whether the synthetically generated data has enough resemblance to the real-world images to improve the performance of computer vision models in practice. We present experiments assessing the effectiveness on real-world data of systems trained on synthetic RGB images that are extracted from a video game. We collected over 60000 synthetic samples from a modern video game with similar conditions to the real-world CamVid and Cityscapes datasets. We provide several experiments to demonstrate that the synthetically generated RGB images can be used to improve the performance of deep neural networks on both image segmentation and depth estimation. These results show that a convolutional network trained on synthetic data achieves a similar test error to a network that is trained on real-world data for dense image classification. Furthermore, the synthetically generated RGB images can provide similar or better results compared to the real-world datasets if a simple domain adaptation technique is applied. Our results suggest that collaboration with game developers for an accessible interface to gather data is potentially a fruitful direction for future work in computer vision.

Session

Posters 1

Files

PDF iconExtended Abstract (PDF, 126K)
PDF iconPaper (PDF, 1M)
ZIP iconSupplemental Materials (ZIP, 2M)

DOI

10.5244/C.30.26
https://dx.doi.org/10.5244/C.30.26

Citation

Alireza Shafaei, James Little and Mark Schmidt. Play and Learn: Using Video Games to Train Computer Vision Models. In Richard C. Wilson, Edwin R. Hancock and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 26.1-26.13. BMVA Press, September 2016.

Bibtex

        @inproceedings{BMVC2016_26,
        	title={Play and Learn: Using Video Games to Train Computer Vision Models},
        	author={Alireza Shafaei, James Little and Mark Schmidt},
        	year={2016},
        	month={September},
        	pages={26.1-26.13},
        	articleno={26},
        	numpages={13},
        	booktitle={Proceedings of the British Machine Vision Conference (BMVC)},
        	publisher={BMVA Press},
        	editor={Richard C. Wilson, Edwin R. Hancock and William A. P. Smith},
        	doi={10.5244/C.30.26},
        	isbn={1-901725-59-6},
        	url={https://dx.doi.org/10.5244/C.30.26}
        }