Title: Learning to Interpret and Generate Instructional Recipies
Advisors: Yejin Choi and Luke Zettlemoyer
Supervisory Committee: Yejin Choi (co-Chair), Luke Zettlemoyer (co-Chair), Gina-Anne Levow (GSR, Linguistics), and Hannaneh Hajishirzi (EE)
Abstract: Enabling computers to interpret and generate instructional language has become increasingly important to our everyday lives: we ask our smartphones to set reminders and send messages; we rely on navigation systems to direct us to our destinations. We define instructional recipes as a special case of instructional language, where completion of the instructions results in a goal object. Some examples include cooking recipes, craft projects, and assembly instructions. Developing systems that automatically analyze and generate instructional recipes requires finding solutions to many semantic challenges, such as identifying implicit arguments (e.g., “Bake for 15 min.”) and learning physical attributes of entities (e.g., which ingredients are considered “dry”). Amassing this information has previously relied upon high-cost annotation efforts. We present a pair of models that can interpret and generate instructional recipes, respectively, and are trained on large corpora with minimal supervision — only identification of the goal (e.g., dish to make), list of materials (e.g., ingredients to use), and recipe text. Our interpretation model is a probabilistic model that (1) identifies the sequence of actions described by the text of an instructional recipe and (2) in which of those actions the provided materials (e.g., ingredients) and entities generated by previous actions (e.g., the mixture created by ``Combine flour and sugar") are used. Our generation model generates instructional recipes that create a specified goal (e.g., dish to make) using a set of materials (e.g., ingredients to use). This model uses a novel neural architecture, the neural checklist model, that enables the generation of globally coherent text. Experiments show that our models can successfully be trained to interpret and generate instructional recipes from this unannotated text, while at the same time learning interpretable domain-knowledge.