Context Matters: Refining Object Detection in Video with Recurrent Neural Networks

Subarna Tripathi, Zachary Lipton, Serge Belongie and Truong Nguyen

Abstract

Given the vast amounts of video available online, and recent breakthroughs in object detection with static images, object detection in video offers a promising new frontier. However, motion blur and compression artifacts cause substantial frame-level variability, even in videos that appear smooth to the eye. Additionally, video datasets tend to have sparsely annotated frames. We present a new framework for improving object detection in videos that captures temporal context and encourages consistency of predictions. First, we train a pseudo-labeler, that is, a domain-adapted convolutional neural network for object detection. The pseudo-labeler is first trained individually on the subset of labeled frames, and then subsequently applied to all frames. Then we train a recurrent neural network that takes as input sequences of pseudo-labeled frames and optimizes an objective that encourages both accuracy on the target frame and consistency across consecutive frames. The approach incorporates strong supervision of target frames, weak-supervision on context frames, and regularization via a smoothness penalty. Our approach achieves mean Average Precision (mAP) of 68.73, a 7.1 improvement over the strongest image-based baselines for the Youtube-Video Objects dataset. Our experiments demonstrate that neighboring frames can provide valuable information, even absent labels.

Session

Posters 1

Files

Extended Abstract (PDF, 516K)
Paper (PDF, 410K)

DOI

10.5244/C.30.44
https://dx.doi.org/10.5244/C.30.44

Citation

Bibtex

        @inproceedings{BMVC2016_44,
        	title={Context Matters: Refining Object Detection in Video with Recurrent Neural Networks},
        	author={Subarna Tripathi, Zachary  Lipton, Serge Belongie and Truong Nguyen},
        	year={2016},
        	month={September},
        	pages={44.1-44.12},
        	articleno={44},
        	numpages={12},
        	booktitle={Proceedings of the British Machine Vision Conference (BMVC)},
        	publisher={BMVA Press},
        	editor={Richard C. Wilson, Edwin R. Hancock and William A. P. Smith},
        	doi={10.5244/C.30.44},
        	isbn={1-901725-59-6},
        	url={https://dx.doi.org/10.5244/C.30.44}
        }