Realtime Novel View Synthesis with Eigen-Texture Regression
Yuta Nakashima, Fumio Okura, Norihiko Kawai, ryosuke Kimura, Hiroshi Kawasaki, Katsushi Ikeuchi and Ambrosio Blanco
Abstract
Realtime novel view synthesis, which generates a novel view of a real object or scene
in realtime, enjoys a wide range of applications including augmented reality, telepresence, and immersive telecommunication. Image-based rendering (IBR) with rough geometry can be done using only an off-the-shelf camera and thus can be used by many
users. However, IBR from images in the wild (e.g., lighting condition changes or the
scene contains objects with specular surfaces) has been a tough problem due to color discontinuity; IBR with rough geometry picks up appropriate images for a given viewpoint,
but the image used for a rendering unit (a face or pixel) switches when the viewpoint
moves, which may cause noticeable changes in color. We use the eigen-texture technique, which represents images for a certain face using a point in the eigenspace. We
propose to regress a new point in this space, which moves smoothly, given a viewpoint
so that we can generate an image whose color smoothly changes according to the point.
Our regressor is based on a neural network with a single hidden layer and hyperbolic
tangent nonlinearity.
Session
Posters
Files
Paper (PDF)
Supplementary (PDF)
DOI
10.5244/C.31.83
https://dx.doi.org/10.5244/C.31.83
Citation
Yuta Nakashima, Fumio Okura, Norihiko Kawai, ryosuke Kimura, Hiroshi Kawasaki, Katsushi Ikeuchi and Ambrosio Blanco. Realtime Novel View Synthesis with Eigen-Texture Regression. In T.K. Kim, S. Zafeiriou, G. Brostow and K. Mikolajczyk, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 83.1-83.12. BMVA Press, September 2017.
Bibtex
@inproceedings{BMVC2017_83,
title={Realtime Novel View Synthesis with Eigen-Texture Regression},
author={Yuta Nakashima, Fumio Okura, Norihiko Kawai, ryosuke Kimura, Hiroshi Kawasaki, Katsushi Ikeuchi and Ambrosio Blanco},
year={2017},
month={September},
pages={83.1-83.12},
articleno={83},
numpages={12},
booktitle={Proceedings of the British Machine Vision Conference (BMVC)},
publisher={BMVA Press},
editor={Tae-Kyun Kim, Stefanos Zafeiriou, Gabriel Brostow and Krystian Mikolajczyk},
doi={10.5244/C.31.83},
isbn={1-901725-60-X},
url={https://dx.doi.org/10.5244/C.31.83}
}