Computer vision enhanced lightweight system for interface automation (CELSIA)

Abstract

This paper presents a novel approach to automatically detecting and interacting with contours on desktop screenshots and managing and generating responses for a chat application using a database of past messages. Our system uses machine learning techniques to classify regions of interest (ROIs) within the screenshot and generate color-coded visualizations of the detected contours. This program is a base for letting programs understand which part on the screen means what and can help hint at what kind of contour the program is encountering. Additionally, the approach uses natural language processing techniques such as the transformers library and annoy to extract keywords from messages and build an index for efficient searching and retrieval of relevant messages. By building the library for running the program on an individual sequential language processor, we are able to perform automatic interface automation when details including position and information of the page are given on any device. However, concluded from the experiences, it requires more optimization to ensure the program can run as lightweight to portable laptop chips. Half the load of all the detecting systems decreases the processing speed and the accuracy of the program at the start of the experiment. While a full load experiment is not performed in this research without manual assisting, we predicted and concluded from our current data that it is possible to do so by using the same code structure as now.