Answering visual what-if questions: from actions to predicted scene descriptions

Misha Wagner, Hector Basevi, Rakshith Shetty, Wenbin Li, Mateusz Malinowski, Mario Fritz, Ales Leonardis

Research output: Chapter in Book/Report/Conference proceedingConference contribution

242 Downloads (Pure)

Abstract

In-depth scene descriptions and question answering tasks have greatly increased the scope of today's definition of scene understanding. While such tasks are in principle open ended, current formulations primarily focus on describing only the current state of the scenes under consideration. In contrast, in this paper, we focus on the future states of the scenes which are also conditioned on actions. We posit this as a question answering task, where an answer has to be given about a future scene state, given observations of the current scene, and a question that includes a hypothetical action. Our solution is a hybrid model which integrates a physics engine into a question answering architecture in order to anticipate future scene states resulting from object-object interactions caused by an action. We demonstrate first results on this challenging new problem and compare to baselines, where we outperform fully data-driven end-to-end learning approaches.
Original languageEnglish
Title of host publicationComputer Vision – ECCV 2018 Workshops
Subtitle of host publicationMunich, Germany, September 8-14, 2018, Proceedings
EditorsLaura Leal-Taixé, Stefan Roth
PublisherSpringer
Pages521-537
Number of pages17
ISBN (Electronic)978-3-030-11009-3
ISBN (Print)978-3-030-11008-6
DOIs
Publication statusE-pub ahead of print - 23 Jan 2019
Event Visual Learning and Embofdied Agents in Simulation Environment Workshop at 15th European Conference on Computer Vision (ECCV 2018) - Munich, Germany
Duration: 9 Sept 20189 Sept 2018

Publication series

NameLecture Notes in Computer Science
Volume11129
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference Visual Learning and Embofdied Agents in Simulation Environment Workshop at 15th European Conference on Computer Vision (ECCV 2018)
Country/TerritoryGermany
CityMunich
Period9/09/189/09/18

Keywords

  • Scene understanding
  • Visual Turing Test
  • Visual question answering
  • Intuitive physics

Fingerprint

Dive into the research topics of 'Answering visual what-if questions: from actions to predicted scene descriptions'. Together they form a unique fingerprint.

Cite this