Image Captioning with Sentiment Terms via Weakly-Supervised Sentiment Dataset

Andrew Shin, Yoshitaka Ushiku and Tatsuya Harada

Abstract

Image captioning task has become a highly competitive research area with successful application of convolutional and recurrent neural networks, especially with the advent of long short-term memory (LSTM) architecture. However, its primary focus has been a factual description of the images, including the objects, movements, and their relations. While such focus has demonstrated competence, describing the images along with non-factual elements, namely sentiments of the images expressed via adjectives, has mostly been neglected. We attempt to address this issue by fine-tuning an additional convolutional neural network solely devoted to sentiments, where dataset on sentiment is built upon a data-driven, multi-label approach. Our experimental results show that our method can generate image captions with sentiment terms that are more compatible with the images than solely relying on features devoted to object classification, while capable of preserving the semantics.

Session

Posters 1

Files

Extended Abstract (PDF, 1M)
Paper (PDF, 6M)
Supplemental Materials (ZIP, 7M)

DOI

10.5244/C.30.53
https://dx.doi.org/10.5244/C.30.53

Citation

Bibtex

        @inproceedings{BMVC2016_53,
        	title={Image Captioning with Sentiment Terms via Weakly-Supervised Sentiment Dataset},
        	author={Andrew Shin, Yoshitaka Ushiku and Tatsuya Harada},
        	year={2016},
        	month={September},
        	pages={53.1-53.12},
        	articleno={53},
        	numpages={12},
        	booktitle={Proceedings of the British Machine Vision Conference (BMVC)},
        	publisher={BMVA Press},
        	editor={Richard C. Wilson, Edwin R. Hancock and William A. P. Smith},
        	doi={10.5244/C.30.53},
        	isbn={1-901725-59-6},
        	url={https://dx.doi.org/10.5244/C.30.53}
        }