Abstract

Approaches to Grounded Language Learning are commonly focused on a single task-based final performance measure which may not depend on desirable properties of the learned hidden representations, such as their ability to predict object attributes or generalize to unseen situations. To remedy this, we present GroLLA, an evaluation framework for Grounded Language Learning with Attributes based on three sub-tasks: 1) Goal-oriented evaluation; 2) Object attribute prediction evaluation; and 3) Zero-shot evaluation. We also propose a new dataset CompGuessWhat?! as an instance of this framework for evaluating the quality of learned neural representations, in particular with respect to attribute grounding. To this end, we extend the original GuessWhat?! dataset by including a semantic layer on top of the perceptual one. Specifically, we enrich the VisualGenome scene graphs associated with the GuessWhat?! images with several attributes from resources such as VISA and ImSitu. We then compare several hidden state representations from current state-of-the-art approaches to Grounded Language Learning. By using diagnostic classifiers, we show that current models' learned representations are not expressive enough to encode object attributes (average F1 of 44.27). In addition, they do not learn strategies nor representations that are robust enough to perform well when novel scenes or objects are involved in gameplay (zero-shot best accuracy 50.06%).

Highlights

  • Several grounded language learning tasks have been proposed to capture perceptual aspects of language (Shekhar et al, 2017; Hudson and Manning, 2019; Suhr et al, 2019; Agrawal et al, 2018)

  • Guesser accuracy We evaluate the GDSE and DeVries models in gameplay mode using the set of reference games provided in CompGuessWhat?!

  • Attribute Prediction We use the CompGuessWhat?! benchmark to compare several dialogue state representations: DeVries-SL: the representation learned by the Questioner model presented in that generates the question tokens conditioned on the image features and is trained using Supervised Learning (SL)

Read more

Summary

Introduction

Several grounded language learning tasks have been proposed to capture perceptual aspects of language (Shekhar et al, 2017; Hudson and Manning, 2019; Suhr et al, 2019; Agrawal et al, 2018). In is appliance has buttons is white microwave left-of Turn Question. Legend abstract situated oven is appliance has buttons is white the literature, several methods have been proposed to analyse what kind of information is captured by neural network representations (Kadar et al, 2017; Belinkov and Glass, 2019). Most of these works examine the hidden state representations learned by models trained on only textual data. Investigating whether the representations learned by a model exhibit forms of attribute composition is beneficial for assessing model interpretability and generalisation

Objectives
Results
Conclusion

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.