Abstract
<span>This paper proposes a way to control home appliances using a multimodal interaction system such as speech, gestures, and smartphone applications. The sensor to capture speech, in the Indonesian language, and gestures from users are Kinect v2. Speech recognition process with the Google Cloud Speech, gesture recognition process with the K-Means clustering, and dialogue system process with the finite state machine. Users can also use the smartphone application to remotely control home appliances through mobile devices such as tablets or smartphones that are connected directly to the real-time database. There are two output responses from this system, namely the audio response generator to provide feedback to the user through the sound of the computer speaker and also provide an action to control home appliances use Esp8266. The average level of accuracy testing of interaction using dialogue systems and gesture are 92.5% and 79,25%. Interaction using dialogue systems is better than gesture. Smartphone applications can control home appliances properly.</span>
Highlights
Humans communicate with each other do depend on speech but they use different modes or ways, such as gestures, hand expressions, facial expressions, touch screen, keyboard or pointing device [1]
The novelty of this research is the multimodal interaction algorithm using a smartphone application, speech, and gesture recognition that is equipped with a dialogue system so that machines can interact with humans
Home automation can use speech control systems where speech is converted to text using automatic speech recognition such as the Microsoft Speech API or Google Cloud Speech API, send to the server using Wi-Fi
Summary
Humans communicate with each other do depend on speech but they use different modes or ways, such as gestures, hand expressions (sign language), facial expressions (gaze/eye movements), touch screen, keyboard or pointing device [1]. Smart home technology offers a new opportunity to improve the comfort of people with computing technology that provides enhanced communication through a variety from multimodal inputs speech, gesture and mobile application. This communication translates into actions that help the smart home system to complete the. Multimodal inputs used are speech, gestures, and smartphone applications, so make it easy for users to control all their home appliances. This smart home system is equipped with a dialogue system so humans can interact to communicate their intents, such as to control room temperature and lights. The novelty of this research is the multimodal interaction algorithm using a smartphone application, speech, and gesture recognition that is equipped with a dialogue system so that machines can interact with humans
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: International Journal of Interactive Mobile Technologies (iJIM)
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
