Enhancing Alexa Skill Testing Through Improved Utterance Discovery

Abstract

Extracting skill utterances is a crucial step in testing and evaluating smart speaker skills. Previous works have proposed various techniques for extracting utterances from the skill web page to test and evaluate smart speaker skills. In this paper, we evaluate the effectiveness and correctness of different utterance-extracting techniques proposed in previous works. Our evaluation reveals that some techniques can capture incorrect utterances that the skill will not accept as spoken utterances. We also find that all the proposed techniques would never capture the total utterances supported by skills, and combining these techniques yields the best results in terms of text parsing. To address these limitations, we propose a new technique that combines the strengths of previous techniques and leverages human to interact with a small set of skills to expand the coverage for testing other skills. We evaluate the effectiveness of our technique and demonstrate that it can capture a higher number of utterances supported by skills. We delved into the impact assessment of utterance extraction, aiming to enhance the thoroughness and effectiveness of skill testing. We conducted an impact study on 11 skills to assess the importance of utterance extraction in the context of skills testing. Our findings demonstrate that our technique captures a higher number of utterances compared to previous techniques. Through our evaluation, we provide insights into the significance of discovering more utterances in the testing context, and demonstrate the effectiveness of our proposed technique in capturing more utterances for skill testing and evaluation.