Abstract

We present a system, CRUISE, that guides ordinary software developers to build a high quality natural language understanding (NLU) engine from scratch. This is the fundamental step of building a new skill in personal assistants. Unlike existing solutions that require either developers or crowdsourcing to manually generate and annotate a large number of utterances, we design a hybrid rule-based and data-driven approach with the capability to iteratively generate more and more utterances. Our system only requires light human workload to iteratively prune incorrect utterances. CRUISE outputs a well trained NLU engine and a large scale annotated utterance corpus that third parties can use to develop their custom skills. Using both benchmark dataset and custom datasets we collected in real-world settings, we validate the high quality of CRUISE generated utterances via both competitive NLU performance and human evaluation. We also show the largely reduced human workload in terms of both cognitive load and human pruning time consumption.

Highlights

  • Intelligent voice-enabled personal assistants have been emerging in our daily life, such as Alexa, Google Assistant, Siri, Bixby, etc

  • The key of developing a new skill is to understand all varieties of user utterances and carry out the intent of users, referred to as natural language understanding (NLU) engine

  • Since users intend to use spoken language to interact with personal assistant agents, most industrial products are focused on spoken language understanding (SLU) in which it is sufficient to understand user query by classifying the intent and identifying a set of slots (Liu and Lane, 2016)

Read more

Summary

Introduction

Intelligent voice-enabled personal assistants have been emerging in our daily life, such as Alexa, Google Assistant, Siri, Bixby, etc. Researches have been made to bootstrap the utterance generations These approaches first generate canonical utterances based on either lexicon/grammar (Wang et al, 2015) or language/SQL templates (Iyer et al, 2017); utilize crowdsourcing to create paraphrases and correct labels. They require software developers to have natural language expertise and still heavily rely on costly crowdsourcing. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics-System Demonstrations, pages 105–110 Melbourne, Australia, July 15 - 20, 2018. c 2018 Association for Computational Linguistics such, CRUISE does not depend on crowdsourcing to conduct the heavy task of manually generating utterances and annotating slots

Background and Related Work
Our Settings
CRUISE Design
Rule-based Iteration Design
Data-driven Tagged Sentence Filler
Experimental Evaluation
Objective Evaluation via NLU Engines
Subjective Human Evaluation
Findings
Human Workload Analysis
Full Text
Paper version not known

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.