PROCESSAMENTO DE LINGUAGEM NATURAL PARA RECONHECIMENTO DE ENTIDADES NOMEADAS EXTRAÍDAS DE CORPUS

Abstract

This article presents a code implementation to automate the reading, analysis and generation of summaries of unstructured web texts, highlighting the crucial role of Natural Language Processing (NLP). The process was divided into two stages: data collection and pre-processing. In the first stage, a scientific article was selected and relevant data was extracted via web scraping, organized into an HTML page hosted online. In pre-processing, tokenization, normalization, stopword removal and word counting were performed. Using NLTK, the most important sentences were identified and ranked based on keyword frequency, allowing the selection of the most relevant sentences from the introduction, methodology, results/dicussion and conclusion sections. Discrepancies such as links and figure descriptions have been removed to improve the clarity of the abstract. The “most_common()” method was used to select the most relevant words in each section. After additional processing to delete unnecessary words, the final summary was clear and understandable, despite minor flaws. This automated approach provides an efficient way to synthesize information from scientific articles, optimizing the textual analysis process.