WordsEye: A Text-to-Scene Conversion System

Abstract

I will present WordsEye, a natural language understanding system that generates three-dimensional scenes from English descriptions of those scenes (joint work with Bob Coyne, Chris Johnson, Owen Rambow, Srinivas Bangalore).

WordsEye works by first parsing the input, using Church’s (1988) part of speech tagger and a version of Michael Collins’ (1999) parser. The parser output is converted into a dependency structure, and semantic rules are then applied to produce a high-level scene description. This scene description is then fed into a set of depiction rules that decide what objects to use, how to arrange them, and in the case of human figures, how to pose them depending upon the action being depicted. WordsEye also does reference resolution, which is obviously crucial for deciding whether a just-named object is in fact the same as an object previously named in the discourse.

The talk will give an overview of the current state of WordsEye, detailing the natural language processing component and the depiction component. I will show how WordsEye depicts spatial relations, actions, attributes, properties of the environment and how it does rudimentary common-sense reasoning to resolve potential conflicts in depiction. The talk will also discuss possible applications of this technology, as well as future directions for research in this area.