Unstructured Data Processing Research Articles

An Extended Pattern Based Comprehensive Stemmer for the Urdu Language

The Urdu language is used by approximately 200 million people for spoken and written communications on a daily basis. There is a substantial amount of unstructured Urdu textual data that is available worldwide. Data mining techniques can be used to extract meaningful knowledge from such a large, potentially informative source of data. There are many text processing systems available to process unstructured textual data. However, these systems are mostly language specific and developed for a variety of languages such as English, Spanish, Chinese, etc. Unfortunately, there are not as many language processing resources available for Urdu. Stemming is one of the most important preprocessing steps in the text mining process and its goal is to reduce grammatical words form, e.g., parts of speech, gender, tense, and so on, to their root form. In this work, we have extended the stemming capabilities of our existing pattern-based comprehensive stemming system for Urdu text. In addition to the existing stemming rules in previous work, we introduce novel stemming rules for prefix, and infix stemming. We also optimize the existing suffix removal rules and extend the add character lists for word normalization. These stemming rules are generic and have the ability to generate the stem of Urdu words as well as loan words (words belonging to other languages i.e. Arabic, Persian, Turkish, etc). In the experimental evaluation, we have observed a significant improvement in the overall stemming accuracy of our proposed pattern-based Urud stemmer, which demonstrates the adoptability of the proposed stemming approach for a variety of text-processing applications.

Energy system optimization based on fuzzy decision support system and unstructured data

To address the complex challenges in energy systems, this study proposes a novel optimization framework that integrates fuzzy decision support and unstructured data processing technologies. This framework aims to improve efficiency, reduce costs, decrease environmental impact, increase system flexibility, and enhance user satisfaction, thereby promoting sustainable development in the energy industry. The framework combines the innovative Energy Semantic Mapping Model (ESMM) and the advanced deep learning architecture ResNet to process textual and visual data effectively. ESMM enables accurate prediction of energy demand, while ResNet significantly reduces equipment maintenance costs and improves energy distribution efficiency. These advancements are critical as they address the limitations of existing approaches in handling large-scale unstructured data and making informed decisions under uncertainty. The Environmental Impact Assessment (EIA) confirms the model's effectiveness in reducing carbon emissions. A comprehensive economic analysis demonstrates substantial cost savings in energy procurement and operations and maintenance, with overall savings exceeding 25%. Enhanced user satisfaction and reduced system response times further validate the practical utility of the proposed approach. Additionally, a genetic algorithm is used to optimize the fuzzy rule base, enhancing the robustness and adaptability of the model. Experimental results show superior performance compared to traditional systems, providing strong empirical evidence for the intelligent transformation of energy systems. This research contributes to the field by offering a more sophisticated and flexible solution for managing energy systems, particularly in terms of leveraging unstructured data and improving decision-making processes.

Design of software system architecture for reading and publishing amateur literary works

The article discusses the functionality and key requirements for a web application for reading and publishing amateur literary works. Against the backdrop of current changes in the book industry caused by the development of technology and the emergence of self-publishing, the need to create platforms that provide authors with the opportunity to publish content independently is emphasized. One of the main problems that authors face when publishing is the loss of the original formatting of their works when transferring them to text editors on web platforms. This paper proposes a solution to this problem by developing a web application with the function of importing the content of text files while preserving their original formatting. The article also contains a comparative analysis of popular web-based platforms for publishing works, such as Archive of Our Own, Wattpad, and Arkush, which helped to highlight the advantages and disadvantages of each of them. In particular, the difficulties associated with storing and processing unstructured data, such as images and text files, were investigated. To solve the problem of storing such data, a comparative analysis of three approaches was conducted: storing data in relational databases, local file systems, and cloud storage. Taking into account the results of this analysis, the web application under development was chosen to integrate with the Amazon Web Services cloud provider and use Amazon S3 cloud storage, which provides scalability, reliability, and convenience in managing large amounts of unstructured data. It was also decided to use the Amazon CloudFront CDN service to improve the speed of data access and increase overall system performance. In addition, the article describes the choice of a technology stack for implementing the web application, which includes Java, Spring Framework, MySQL, ORM Hibernate for the server side and HTML, CSS, React.js for the client side. The article highlights important aspects of designing a web application architecture aimed at scalability and resilience to large amounts of data and an increase in the number of active users, and provides a basis for further system improvement.

MAPRS: An intelligent approach for post-prescription review based on multi-label learning

Antimicrobial resistance (AMR) is a major threat to public health worldwide. It is a promising way to improve appropriate prescription by the review and stewardship of antimicrobials, and Post-Prescription Review (PPR) is currently the main tool used in hospitals. Existing methods of PPR typically focus on the dichotomy of antimicrobial prescription based on binary classification which, however, is usually a multi-label classification problem. Moreover, previous research did not explain the causes beneath the inappropriate antimicrobial used in the clinical setting, which could be practically important for problem location and decision improvement. In this paper, we collected antimicrobial prescriptions and related data from clean surgery in a hospital in northeastern China, and proposed a Multi-label Antimicrobial Post-Prescription Review System (MAPRS). MAPRS first uses NLP techniques to process unstructured data in prescriptions and explores the value of clinical record text for solving medical problems. Then, Classifier Chains are used to deal with multi-label problems and fused with machine learning algorithms to construct a classifier. At last, a SHAP explanation module is introduced to explain the inappropriate prescriptions. The experimental results show that MAPRS could achieve great performance in a challenging six-category multi-label task, with a subset accuracy of 90.7 % and an average AUROC of 94.3 %. Our results can help hospitals to perform intelligent prescription review and improve the antimicrobial stewardship.

A Deep Learning-Based Method for Preventing Data Leakage in Electric Power Industrial Internet of Things Business Data Interactions.

In the development of the Power Industry Internet of Things, the security of data interaction has always been an important challenge. In the power-based blockchain Industrial Internet of Things, node data interaction involves a large amount of sensitive data. In the current anti-leakage strategy for power business data interaction, regular expressions are used to identify sensitive data for matching. This approach is only suitable for simple structured data. For the processing of unstructured data, there is a lack of practical matching strategies. Therefore, this paper proposes a deep learning-based anti-leakage method for power business data interaction, aiming to ensure the security of power business data interaction between the State Grid business platform and third-party platforms. This method combines named entity recognition technologies and comprehensively uses regular expressions and the DeBERTa (Decoding-enhanced BERT with disentangled attention)-BiLSTM (Bidirectional Long Short-Term Memory)-CRF (Conditional Random Field) model. This method is based on the DeBERTa (Decoding-enhanced BERT with disentangled attention) model for pre-training feature extraction. It extracts sequence context semantic features through the BiLSTM, and finally obtains the global optimal through the CRF layer tag sequence. Sensitive data matching is performed on interactive structured and unstructured data to identify privacy-sensitive information in the power business. The experimental results show that the F1 score of the proposed method in this paper for identifying sensitive data entities using the CLUENER 2020 dataset reaches 81.26%, which can effectively prevent the risk of power business data leakage and provide innovative solutions for the power industry to ensure data security.

Analysis of methods and algorithms for processing unstructured text data based on JSON technology

The object of research is the process of automating systems for structuring data from several sources. The subject of the research is methods and algorithms for implementing a complete system for automated and parallel processing, validation and structuring of data. One of the most problematic areas is the merging of databases with different structures and several common fields into a generalized structure. The research was aimed at developing a system to increase the efficiency of automation of big data processing. As a result of the work, optimization methods were studied, the influence of their internal parameters on the operation of algorithms was analyzed, their main advantages and disadvantages were determined, and software was developed in which the corresponding methods were implemented. An algorithm for structuring data before processing has been obtained. Data structuring is achieved by performing the «mapping» operation. Mapping can take place by indexes of already cleaned data or using a defined dictionary with a given set of keys, which allows not to care about the sequence of storing values and their possible shift. The practical significance of the developed system lies in the improvement of methods of collecting and processing information for the purpose of its further validation, cleaning and accumulation in the following categories: geographic addresses and geo-coordinates, validation and automated addition of a mobile phone number to the international format, processing of car numbers (in modern and outdated format), VIN code of the engine and car brand, validation of urls of social networks, passport data and processing of personal data. Compared to similar methods for processing large volumes of data, the possibility of splitting the input file or stream into separate parts was used, the cleaned data from which is combined at the end of the system operation. Thanks to this, it is possible to process data whose size exceeds the available volume of the device's RAM, and the method of working with loosely structured text files in CSV format has been improved.

Image Text Extraction and Natural Language Processing of Unstructured Data from Medical Reports

This study presents an integrated approach for automatically extracting and structuring information from medical reports, captured as scanned documents or photographs, through a combination of image recognition and natural language processing (NLP) techniques like named entity recognition (NER). The primary aim was to develop an adaptive model for efficient text extraction from medical report images. This involved utilizing a genetic algorithm (GA) to fine-tune optical character recognition (OCR) hyperparameters, ensuring maximal text extraction length, followed by NER processing to categorize the extracted information into required entities, adjusting parameters if entities were not correctly extracted based on manual annotations. Despite the diverse formats of medical report images in the dataset, all in Russian, this serves as a conceptual example of information extraction (IE) that can be easily extended to other languages.

Development of Classification Method for Lecturer Area of Expertise Based on Scientific Publication Using BERT

Implementing the Artificial Intelligence concept in higher education can be utilized in the context of Human Resource (HR) talent management. The lecturer portfolio provided by the Integrated Resource Information System (SISTER DIKTI) is expected to give an overview of the profiles of all lecturers and map competencies based on groups of fields of knowledge. However, the map of scientific fields based on SISTER data currently available is still subjective. The data is in the form of a group of lecturers' chosen fields of science, independently selected by each lecturer to recognize their expertise. This study discusses the problem of processing unstructured SISTER data. It looks for mapping solutions and classification methods by developing a strategy for classifying groups of scientific fields from unstructured data input. It is necessary to identify the suitability of the chosen field of science compared to that developed through the tri-dharma through identification based on a mapping of the group of fields that can be extracted from the tri-dharma activity, in this case, research represented by scientific publications recorded on SISTER. Therefore, we need an appropriate model to measure similarity, which can then be classified based on abstract documents and scientific publication titles for the classification of scientific fields using NLP based on classification run on DGX A100. This study aims to develop a classification method from titles and abstracts. Scientific publications contained in SISTER are unstructured data, so a corpus is formed to identify the lecturer's field of science. The results show that the classification method developed in this study can measure the similarity of lecturer publications based on abstracts and titles through a vector formation process based on bidirectional encoders and also produces a deep learning model to classify 24 categories of fields of science with an accuracy of 95,0345 percent on training data and 92.876 percent on the test data.

Machine Learning-based Distributed Big data Analytics Framework for IoT Applications

The emerging trend of the Internet of Things (IoT) leads to more real-time applications and it raises the accumulation of more structured and unstructured data. The processing of unstructured and structured data for different distributed applications becomes arduous. Moreover, the accuracy, load balancing, and latency of the services are also challenging. Some of the state-of-art works failed to achieve those parameters. In context with these, we proposed a machine learning-based novel approach that utilizes an SVM classifier. The SVM classifier can be used for the classification and data analytic purposes of features extracted from the data processing step. Data processing employs two steps such as data extraction and data scaling. The data extraction is performed by the adoption of the Principal Component Discriminant power-based Linear Discriminant analysis (PCDP-LDA) technique. The big data framework of the proposed model is evaluated using a tool called Weka. Meanwhile, the data scaling is performed by the Naïve Bayes approach and divides the extracted features into blocks. Experimental analysis is performed and compared with existing approaches. Our proposed approach provides more effective classification and data analytic accuracy than the other approaches. Our approach also provides better latency, and load balancing of data in the distributed big data analysis.

Fully optically controlled Li-ion-mediated artificial vision reflection arc system

Artificial vision systems have been extensively reported for processing unstructured data, performing pattern recognition, and performing visual simulations. Photonic synapses have become the core of most artificial vision systems due to their advantages such as low energy consumption and ultra-fast signal transmission. However, their integration is difficult, the integration difficulty of photon synapses is higher than that of crossbar arrays. Moreover, due to the hybrid drive of light and external power supply, low energy consumption and weight update under full optical control cannot be achieved. Here, we designed a fully light-controlled multifunctional artificial vision reflex arc system. The solar cell and oscillator generate presynaptic voltage signals that are then transmitted to a lithium-ion-based memristor for the stimulation of different wavelengths and light intensities. The results show that our artificial vision system can distinguish wavelength and light intensity information and has good weight update symmetry and repeatability. Finally, the deformation experiment of the shape memory alloy demonstrated that the system can be used for controlling artificial muscles under different light signals.

Machine Learning-based Distributed Big data Analytics Framework for IoT Applications

The emerging trend of the Internet of Things (IoT) leads to more real-time applications and it raises the accumulation of more structured and unstructured data. The processing of unstructured and structured data for different distributed applications becomes arduous. Moreover, the accuracy, load balancing, and latency of the services are also challenging. Some of the state-of-art works failed to achieve those parameters. In context with these, we proposed a machine learning-based novel approach that utilizes an SVM classifier. The SVM classifier can be used for the classification and data analytic purposes of features extracted from the data processing step. Data processing employs two steps such as data extraction and data scaling. The data extraction is performed by the adoption of the Principal Component Discriminant power-based Linear Discriminant analysis (PCDP-LDA) technique. The big data framework of the proposed model is evaluated using a tool called Weka. Meanwhile, the data scaling is performed by the Naïve Bayes approach and divides the extracted features into blocks. Experimental analysis is performed and compared with existing approaches. Our proposed approach provides more effective classification and data analytic accuracy than the other approaches. Our approach also provides better latency, and load balancing of data in the distributed big data analysis.

Mesh-Agnostic Decoders for Supercritical Airfoil Prediction and Inverse Design

Mesh-agnostic models have advantages in terms of processing unstructured spatial data and incorporating partial differential equations. Recently, they have been widely studied for constructing physics-informed neural networks, but they need to be trained on a case-by-case basis and require long training times. On the other hand, fast prediction and design tools are desired for aerodynamic shape designs, and data-driven mesh-based models have achieved great performance. Therefore, this paper proposes a data-driven mesh-agnostic decoder that combines the fast prediction ability of data-driven models and the flexibility of mesh-agnostic models. The model is denoted by an implicit decoder, which consists of two subnetworks, i.e., ShapeNet and HyperNet. ShapeNet is based on implicit neural representation, and HyperNet is a simple neural network. The implicit decoder is trained for the fast prediction of supercritical airfoils. Different activation functions are compared, and a spatial constraint is proposed to improve the interpretability and generalization ability of the model. Then, the implicit decoder is used together with a mesh-based encoder to build a generative model, which is used for the inverse design of supercritical airfoils with specified physical features.

Reconstruction of Radio Environment Map Based on Multi-Source Domain Adaptive of Graph Neural Network for Regression.

The graph neural network (GNN) has shown outstanding performance in processing unstructured data. However, the downstream task performance of GNN strongly depends on the accuracy of data graph structural features and, as a type of deep learning (DL) model, the size of the training dataset is equally crucial to its performance. This paper is based on graph neural networks to predict and complete the target radio environment map (REM) through multiple complete REMs and sparse spectrum monitoring data in the target domain. Due to the complexity of radio wave propagation in space, it is difficult to accurately and explicitly construct the spatial graph structure of the spectral data. In response to the two above issues, we propose a multi-source domain adaptive of GNN for regression (GNN-MDAR) model, which includes two key modules: (1) graph structure alignment modules are used to capture and learn graph structure information shared by cross-domain radio propagation and extract reliable graph structure information for downstream reference signal receiving power (RSRP) prediction task; and (2) a spatial distribution matching module is used to reduce the feature distribution mismatch across spatial grids and improve the model's ability to remain domain invariant. Based on the measured REMs dataset, the comparative results of simulation experiments show that the GNN-MDAR outperforms the other four benchmark methods in accuracy when there is less RSRP label data in the target domain.

INTELLIGENT DATA ANALYSIS ON AN ANALYTICAL PLATFORM

The article discusses methods for processing unstructured data using an analytical platform. The authors analyze existing methods and technologies used to implement data processing and propose new approaches to solving this problem. The possibilities of using analytical platforms to solve the problem of processing source data are considered. The purpose of the article is to explore the possibilities of data import, partial preprocessing, missing data recovery, anomaly removal, spectral processing and noise removal. The authors explored how analytics platforms can function without a data warehouse, obtaining information from any other sources, but the most optimal way is to use them together, and how big data and unstructured data can be processed using an analytics platform. The authors solved a specific problem related to processing problems and proposed ways to solve them using an analytical platform. Particular attention is paid to a complete set of mechanisms that allows you to obtain information from any data source, carry out the entire processing cycle and display the results. Overall, the paper represents an important contribution to the development of raw data processing technologies. The authors plan to continue research in the field of processing big unstructured data.

A Model for Enhancing Unstructured Big Data Warehouse Execution Time

Traditional data warehouses (DWs) have played a key role in business intelligence and decision support systems. However, the rapid growth of the data generated by the current applications requires new data warehousing systems. In big data, it is important to adapt the existing warehouse systems to overcome new issues and limitations. The main drawbacks of traditional Extract–Transform–Load (ETL) are that a huge amount of data cannot be processed over ETL and that the execution time is very high when the data are unstructured. This paper focuses on a new model consisting of four layers: Extract–Clean–Load–Transform (ECLT), designed for processing unstructured big data, with specific emphasis on text. The model aims to reduce execution time through experimental procedures. ECLT is applied and tested using Spark, which is a framework employed in Python. Finally, this paper compares the execution time of ECLT with different models by applying two datasets. Experimental results showed that for a data size of 1 TB, the execution time of ECLT is 41.8 s. When the data size increases to 1 million articles, the execution time is 119.6 s. These findings demonstrate that ECLT outperforms ETL, ELT, DELT, ELTL, and ELTA in terms of execution time.

Student Feedback Mining System Using Sentiment Analysis

In education, the importance of gathering and understanding student feedback cannot be over stated. Traditional methods of gathering feedback through surveys and assessments often suffer from limitations such as low response rates and subjectivity. This paper proposes a new approach to improve the feedback collection process by implementing a Student Feedback Mining System using sentiment analysis. The goal of this system is to automatically analyze and extract sentiment from student feedback and provide educators and institutions with valuable insights into students' overall satisfaction and feelings toward various aspects of their academic experience. The system uses natural language processing (NLP) techniques to process unstructured text data, identify sentiments, and categorize feedback into positive, negative, or neutral feelings. Collecting student feedback through various channels such as online surveys, discussion forums and other feedback mechanisms. Cleaning and preprocessing the collected text data to remove noise, irrelevant information and standardize the format for analysis. Using state of the art NLP algorithms and sentiment analysis techniques to determine the emotional tone of feedback. This includes classifying sentiment as positive, negative, or neutral. Grouping feedback based on specific criteria such as course content, teaching methods, facilities and overall learning experience to provide a comprehensive understanding of the various aspects affecting student satisfaction. Presenting analyzed data through intuitive visualizations and reports, making it easy for educators and administrators to interpret and respond to feedback. Creating feedback that enables continuous improvement by providing actionable insights for educators to improve their teaching methods, curriculum and overall learning environment. The proposed system aims to overcome the limitations of traditional feedback collection methods by automating the analysis process, providing real-time insights and supporting a data-driven approach to improving education. Using sentiment analysis, educators and institutions can proactively solve problems, improve the quality of education, and create a more positive and engaging learning environment.

UDNet: A Unified Deep Learning-Based AutoML Framework to Execute Multiple ML Strategies for Multi-Modal Unstructured Data Processing

UDNet: A Unified Deep Learning-Based AutoML Framework to Execute Multiple ML Strategies for Multi-Modal Unstructured Data Processing

Risk stratification of coronary artery disease using the artificial intelligence-enabled electrocardiogram

Abstract Background Atherosclerotic cardiovascular disease (ASCVD), of which coronary artery disease (CAD) is the primary manifestation, is the leading cause of death worldwide. The AHA/ACC Pooled Cohort Equation (PCE) facilitates risk stratification and initiation of primary prevention of ASCVD, however their performance is suboptimal. Purpose We develop an artificial intelligence-based electrocardiogram (ECG) analysis (AI-ECG) algorithm to detect different stages of CAD and assessed the value of AI-ECG over PCE to predict incident myocardial infarction (MI) and survival over 3-years. Methods Using electronic health record data from approximately 7 million patients seen at over 70 hospitals and clinics across 5 states in the USA, we created independent AI-ECG models to identify elevated coronary artery calcium (CAC) by computer tomography, the presence of obstructive CAD by stress test or angiography, and left ventricular regional akinesis by echocardiogram, presumed to represent prior MI. Deep structured data, AI-enabled natural language processing of unstructured data were used to ensure cohort, and physician review of labeling were used to ensure cohort accuracy. These models were ensembled into a single model that was used to assess the utility of AI-ECG for predicting new occurrence MI in PCE- and age-stratified cohorts of patient with no prior history of ASCVD, over a 3-year timeframe using Cox’s proportional hazards analysis. Results A total of 6,080 patients (15,192 ECGs) were labeled positive for CAC ≥ 300, 12,561 patients (18,915 ECGs) positive for obstructive CAD, and 10,767 patients (23,870 ECGs) positive for regional akinesis. AI-ECG for identifying elevated CACS, obstructive CAD, and regional akinesis achieved AUROCs of 0.86, 0.80, and 0.93, respectively. All AI-ECG models identified increased 3-year risk for MI and death in AI-ECG positive versus AI-ECG negative patients and subgroups with 2 or 3 positive results had stepwise increased risk (Figure A). In a 3-year survival analysis of all-comer patients, the hazard ratio for those that tested positive on 1, 2, or 3 versus 0 positive disease-specific models were 1.61 (1.53-1.69). 2.40 (2.27-2.55), 3.42 (3.21-3.65), respectively. Risk stratification was also observed in cohorts without prior ASCVD stratified by low and high PCE-derived 10-year ASCVD risk (Figure B) and in cohorts stratified by age, including under the age of 40. Conclusions AI-ECG has potential to address unmet need for accessible risk stratification in patients in whom PCE under, over, or insufficiently estimate ASCVD risk, and in whom risk assessment over periods of time shorter than 10 years is desired.

Investigation of Artificial Intelligence algorithms in education

The application of Artificial Intelligence (AI) in education has garnered increasing attention in recent years. This study investigates the primary methods of AI in education, including The Teaching Machine System, Intelligent Tutoring Systems, Intelligent Educational Systems, and the Intelligent College Student Comprehensive Quality Evaluation Model. The Teaching Machine System employs pre-defined rules and algorithms to deliver personalized instruction to students. Intelligent Tutoring Systems use AI algorithms to provide real-time feedback and guidance to students, tailoring the learning experience to address their learning gaps. Intelligent Educational Systems use AI to assist with instructional design, assessment, and delivery, while the Intelligent College Student Comprehensive Quality Evaluation Model is an AI-based system designed to evaluate students' comprehensive quality. Furthermore, this paper provides relevant descriptions and insights into the applications of AI in language and mathematics learning, highlighting the limitations of current AI models in these areas. Despite the advances in AI, challenges still exist, including limitations in processing unstructured data and the need for more effective human-AI interactions. Nonetheless, the potential of AI to enhance the quality and accessibility of education remains promising, and further research is needed to explore its full potential in the field of education.

Spike Optimization to Improve Properties of Ferroelectric Tunnel Junction Synaptic Devices for Neuromorphic Computing System Applications.

The continuous advancement of Artificial Intelligence (AI) technology depends on the efficient processing of unstructured data, encompassing text, speech, and video. Traditional serial computing systems based on the von Neumann architecture, employed in information and communication technology development for decades, are not suitable for the concurrent processing of massive unstructured data tasks with relatively low-level operations. As a result, there arises a pressing need to develop novel parallel computing systems. Recently, there has been a burgeoning interest among developers in emulating the intricate operations of the human brain, which efficiently processes vast datasets with remarkable energy efficiency. This has led to the proposal of neuromorphic computing systems. Of these, Spiking Neural Networks (SNNs), designed to closely resemble the information processing mechanisms of biological neural networks, are subjects of intense research activity. Nevertheless, a comprehensive investigation into the relationship between spike shapes and Spike-Timing-Dependent Plasticity (STDP) to ensure efficient synaptic behavior remains insufficiently explored. In this study, we systematically explore various input spike types to optimize the resistive memory characteristics of Hafnium-based Ferroelectric Tunnel Junction (FTJ) devices. Among the various spike shapes investigated, the square-triangle (RT) spike exhibited good linearity and symmetry, and a wide range of weight values could be realized depending on the offset of the RT spike. These results indicate that the spike shape serves as a crucial indicator in the alteration of synaptic connections, representing the strength of the signals.