Abstract
A lot of services are using short messages for various purposes today, for example, stores are sending promotional offers, and EMERCOM of Russia informs population in the event of a threat of natural and technogenic emergencies. Selecting short texts of the template messages from general traffic could be used to filter spam and mailings, as well as to protect users from fraudulent activities. Such arrays of messages are often reaching such a large size that their storage and processing on a single dedicated personal computer or server becomes practically impossible. This work aims at developing approaches to the efficient identification and clustering of the template texts from large arrays of the short messages using the Apache Spark framework for distributed processing of the unstructured data. Main approaches to identifying templates and clustering textual information are considered. Approaches were developed making it possible to cluster in large arrays of messages using distributed computation without preliminary acquisition of the text vector representations. Algorithms are provided for efficient identification of the template messages from large arrays of short texts. Algorithms were compared in terms of performance and quality of pattern identification
Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: Herald of the Bauman Moscow State Technical University. Series Instrument Engineering
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
