Pre-processing Algorithm Research Articles

A method for optimizing text preprocessing and text classification using multiple cycles of learning with an application on shipbrokers emails

Optimizing text preprocessing and text classification algorithms is an important, everyday task in large organizations and companies and it usually involves a labor-intensive and time-consuming effort. For example, the filtering and sorting of a large number of electronic mails (emails) are crucial to keeping track of the received information and converting it automatically into useful and profitable knowledge. Business emails are often unstructured, noisy, and with many abbreviations and acronyms, which makes their handling a challenging procedure. To overcome those challenges, a two-step classification approach is proposed, along with a two-cycle labeling procedure in order to speed up the labeling process. Every step incorporates a heuristic classification approach to assign emails to predefined classes by comparing several classification and text vectorization algorithms. These algorithms are compared and evaluated using the F1 score and balanced accuracy. The implementation of the proposed algorithm is demonstrated in a shipbroker agent operating in Greece with excellent performance, improving organization and administration while reducing expenses.

Optimal Image Characterization for In-Bed Posture Classification by Using SVM Algorithm

Identifying patient posture while they are lying in bed is an important task in medical applications such as monitoring a patient after a surgical intervention, sleep supervision to identify behavioral and physiological markers, or for bedsore prevention. An acceptable strategy to identify the patient’s position is the classification of images created from a grid of pressure sensors located in the bed. These samples can be arranged based on supervised learning methods. Usually, image conditioning is required before images are loaded into a learning method to increase classification accuracy. However, continuous monitoring of a person requires large amounts of time and computational resources if complex pre-processing algorithms are used. So, the problem is to classify the image posture of patients with different weights, heights, and positions by using minimal sample conditioning for a specific supervised learning method. In this work, it is proposed to identify the patient posture from pressure sensor images by using well-known and simple conditioning techniques and selecting the optimal texture descriptors for the Support Vector Machine (SVM) method. This is in order to obtain the best classification and to avoid image over-processing in the conditioning stage for the SVM. The experimental stages are performed with the color models Red, Green, and Blue (RGB) and Hue, Saturation, and Value (HSV). The results show an increase in accuracy from 86.9% to 92.9% and in kappa value from 0.825 to 0.904 using image conditioning with histogram equalization and a median filter, respectively.

Klasifikasi Gender Berdasarkan Gambar Menggunakan Metode Deep Learning Pada MATLAB

In the present era, machine intelligence, also known as Artificial Intelligence (AI), is demanded not only to execute specific commands but also to recognize, analyze, or even make decisions, thereby providing desired outputs. By harnessing the power of AI, it is anticipated that desired outcomes will be more accurate and goal achievement will be optimized, minimizing losses. With the capabilities of AI in mind, a research study has been conducted on AI's ability to analyze and make decisions based on specific data. In this study, data in the form of images of men and women were utilized. The objective of this research is to analyze the ability of AI, particularly in gender classification. The method employed in designing this system is Deep Learning, with GoogLeNet as the Convolutional Neural Network utilized. In testing, the data accuracy ranged from 61.8% to 100% for the system without training algorithm options and from 97.5% to 100% for the system with training algorithm options. Testing was also carried out on a smaller set of training data and grayscale images, yielding lower accuracy ranges. From this research, it can be concluded that the quantity of training data, image preprocessing, and training algorithm options are crucial indicators for enhancing prediction accuracy.

Study on the Diffusion and Optimization of Sucrose in Gaido Seak Based on Finite Element Analysis and Hyperspectral Imaging Technology.

As a traditional Chinese dish cutting technology process, Gaidao artificially create cuts embedded in the food surface by cutting through it with knife, a process that currently plays an important role in the beef marinating process. And different Gaidao processes directly affect the beef marination flavour and marination efficiency. This study is the first to propose the use of Hyperspectral imaging technology (HSI) combined with finite element analysis to investigate the effect of Gaidao process on the quality of marinated beef. The study was carried out by collecting spectral information of beef marinated with different sucrose concentrations and combining various pre-processing methods and algorithms such as PLS, BiPLS, iPLS, and SiPLS to establish a quantitative model of sucrose concentration in beef, and finally optimizing parameters such as the length, position and number of Gaidao by Finite Element Analysis (FEA), which showed that when marinated with 1.0 mol/m³ sucrose solution, the concentration of sucrose in all tissues in the Gaidao steak reached 0.8 mol/m³ and above, which greatly improved the diffusion effect of the marinade. This work provides new ideas and methods to optimize the beef marinade Gaidao process, which has important practical value and research significance.

Automated osteoporosis classification and T-score prediction using hip radiographs via deep learning algorithm.

Despite being the gold standard for diagnosing osteoporosis, dual-energy X-ray absorptiometry (DXA) is an underutilized screening tool for osteoporosis. This study proposed and validated a controllable feature layer of a convolutional neural network (CNN) model with a preprocessing image algorithm to classify osteoporosis and predict T-score on the proximal hip region via simple hip radiographs. This was a single-center, retrospective study. An image dataset of 3460 unilateral hip images from 1730 patients (age ⩾50 years) was retrospectively collected with matched DXA assessment for T-score for the targeted proximal hip regions to train (2473 unilateral hip images from 1430 patients) and test (497 unilateral hip images from 300 patients) the proposed CNN model. All images were processed with a fully automated CNN model, X1AI-Osteo. The proposed screening tool illustrated a better performance (sensitivity: 97.2%; specificity: 95.6%; positive predictive value: 95.7%; negative predictive value: 97.1%; area under the curve: 0.96) than the open-sourced CNN models in predicting osteoporosis. Moreover, when combining variables, including age, body mass index, and sex as features in the training metric, there was high consistency in the T-score on the targeted hip regions between the proposed CNN model and the DXA (r = 0.996, p < 0.001). The proposed CNN model may identify osteoporosis and predict T-scores on the targeted hip regions from simple hip radiographs with high accuracy, highlighting the future application for population-based opportunistic osteoporosis screening with low cost and high adaptability for a broader population at risk. TMU-JIRB N201909036.

Edge-Based Real-Time Sensor Data Processing for Anomaly Detection in Industrial IoT Applications

The Industrial Internet of Things (IIoT), which uses devices with sensors to provide real-time insights into crucial processes, has completely changed how industries function. However, there are many problems associated with the sheer volume and speed of data created in industrial environments, particularly when it comes to anomaly detection. The development of edge-based real-time sensor data processing techniques was required because traditional cloud-based solutions frequently experience latency problems and privacy issues. This study suggests a novel method for IIoT applications that focuses on processing sensor data at the edge, close to the data source, for anomaly identification. We offer real-time analysis of sensor data without the need for continuous data transfer to the cloud by utilising the processing capabilities of edge devices, such as industrial gateways and embedded systems. To find anomalies in streams of real-time sensor data, our methodology integrates data pre-processing, feature engineering, and machine learning algorithms. This strategy not only lessens the strain on the network's bandwidth but also ensures quick reaction to urgent situations, cutting downtime and boosting operational effectiveness. Proposed system has adaptive learning features that enable it to continuously adjust to altering ambient factors and sensor properties, enhancing the precision of anomaly detection over time. We provide experimental findings that show how our edge-based anomaly detection system performs well in diverse industrial situations. The results show that, while protecting data privacy and minimising latency, our methodology outperforms conventional cloud-based methods in terms of anomaly detection performance.

Automatic Feature Extraction Based on Envelope Analysis and its Application in Rolling-Element Bearing Fault Detection

Bearing fault diagnosis is essential for the maintenance, durability, and reliability of rotating machines. It can minimize economic losses by removing unplanned downtime in the industry due to the failure of rotary machines. In bearing fault detection, developing fault features extraction techniques that can successfully applicable for various fault severity and different operating conditions is still a critical issue. In the current work, the feature extraction technique is a combination between pre-processing algorithms and envelope analysis method. In the pre-processing stage, the autoregressive (AR) model is used to filter the original signal and remove the deterministic vibration sources, as well as maintain the signal representing the condition of the bearing without contaminating noises. Then, the most suitable frequency band is selected based on the spectral kurtosis (SK) analysis. This band contains the signature frequencies of the roller bearing. After that, envelope analysis is employed for detecting faults at different severity. Finally, the features represented the peaks at fundamental fault frequencies are automatically selected from the envelope spectrum. By analyzing all diagnoses results, it is found that the presented method effectively extracts the features at calculated resonance bearing frequencies and proves the significance of the enhancements in a pre-filtering stage in the overall detection performance. Also, it can benefit from these features in the fault classification fields at different speeds because it is independent of speed variation.

Predicting Groundwater Level using Temporal Attention Enhanced Graph Neural Network

A revolutionary approach for groundwater management is essential. To predicting the water levels using for integrating IoT sensors, cloud computing, and advanced data analysis methods. IoT sensors are employed for real-time measurement of groundwater levels, creating a robust dataset. The paper focuses on predicting future groundwater levels, a crucial aspect of sustainable resource management. To enhance predictive accuracy, a preprocessing algorithm, such as Min-Max normalization, is introduced to clean and normalize the collected data, ensuring its reliability. Additionally, a feature extraction algorithm, such as Principal Component Analysis (PCA), is implemented to identify relevant patterns and trends within the dataset, enhancing the efficiency of subsequent analysis. A novel classification algorithm, Spatial-Temporal Graph Convolutional Network, is introduced, enabling the identification of potential groundwater recharge areas. This classification algorithm leverages historical data and extracted features to categorize regions based on their suitability for groundwater renewal. Finally, this research uses a Temporal attention-enhanced graph Neural Network machine learning algorithm to predict groundwater levels in the next few years. This algorithm utilizes the preprocessed data and extracted features, identifying intricate patterns and trends in historical data to generate precise predictions for groundwater levels in the upcoming years.

Extended Analysis of Raman Spectra Using Artificial Intelligence Techniques for Colorectal Abnormality Classification.

Raman spectroscopy (RS) techniques are attracting attention in the medical field as a promising tool for real-time biochemical analyses. The integration of artificial intelligence (AI) algorithms with RS has greatly enhanced its ability to accurately classify spectral data in vivo. This combination has opened up new possibilities for precise and efficient analysis in medical applications. In this study, healthy and cancerous specimens from 22 patients who underwent open colorectal surgery were collected. By using these spectral data, we investigate an optimal preprocessing pipeline for statistical analysis using AI techniques. This exploration entails proposing preprocessing methods and algorithms to enhance classification outcomes. The research encompasses a thorough ablation study comparing machine learning and deep learning algorithms toward the advancement of the clinical applicability of RS. The results indicate substantial accuracy improvements using techniques like baseline correction, L2 normalization, filtering, and PCA, yielding an overall accuracy enhancement of 15.8%. In comparing various algorithms, machine learning models, such as XGBoost and Random Forest, demonstrate effectiveness in classifying both normal and abnormal tissues. Similarly, deep learning models, such as 1D-Resnet and particularly the 1D-CNN model, exhibit superior performance in classifying abnormal cases. This research contributes valuable insights into the integration of AI in medical diagnostics and expands the potential of RS methods for achieving accurate malignancy classification.

A polytime preprocess algorithm for the maximum independent set problem

A polytime preprocess algorithm for the maximum independent set problem

Optimization of Image Processing Algorithms for Character Recognition in Cultural Typewritten Documents

Linked data is used in various fields as a new way of structuring and connecting data. Cultural heritage institutions have been using linked data to improve archival descriptions and facilitate the discovery of information. Most archival records have digital representations of physical artifacts in the form of scanned images that are non-machine-readable. Optical Character Recognition (OCR) recognizes text in images and translates it into machine-encoded text. This article evaluates the impact of image processing methods and parameter tuning in OCR applied to typewritten cultural heritage documents. The approach uses a multi-objective problem formulation to minimize Levenshtein edit distance and maximize the number of words correctly identified with a non-dominated sorting genetic algorithm (NSGA-II) to tune the methods’ parameters. Evaluation results show that parameterization by digital representation typology benefits the performance of image pre-processing algorithms in OCR. Furthermore, our findings suggest that employing image pre-processing algorithms in OCR might be more suitable for typologies where the text recognition task without pre-processing does not produce good results. In particular, Adaptive Thresholding, Bilateral Filter, and Opening are the best-performing algorithms for the theater plays’ covers, letters, and overall dataset, respectively, and should be applied before OCR to improve its performance.

Ionospheric Response on Solar Flares through Machine Learning Modeling

Following solar flares (SF), the abrupt increase in X-radiation and EUV emission generates additional ionization and higher absorption of, e.g., electromagnetic waves in the sunlit hemisphere of the Earth’s ionosphere. The modeling of the ionosphere under solar flares are motivated by new observations with spacecrafts, satellites, and ground-based measurements. The estimation of modeling parameters for the ionospheric D-region during SF events poses a significant challenge, typically requiring a trial-and-error approach. This research presents a machine learning (ML) methodology for modeling the sharpness (β) and reflection height (H′) during SF events occurred from 2008 to 2017. The research methodology was divided into two separate approaches: an instance-based approach, which involved obtaining SF parameters during the peak SF, and a time-series approach, which involved analyzing time-series data during SFs. The findings of the study revealed that the model for the instance-based approach exhibited mean absolute percentage error (MAPE) values of 9.1% for the β parameter and 2.45% for the H′ parameter. The findings from the time-series approach indicated that the model exhibited lower error rates compared to the instance-based approach. However, it was observed that the model demonstrated an increase in β residuals as the predicted β increased, whereas the opposite trend was observed for the H′ parameter. The main goal of the research is to develop an easy-to-use method that provides ionospheric parameters utilizing ML, which can be refined with additional and novel data as well as other techniques for data pre-processing and other algorithms. The proposed method and the utilized workflow and datasets are available at GitHub.

Deep Compressed Sensing based Data Imputation for Urban Environmental Monitoring

Data imputation is prevalent in crowdsensing, especially for Internet of Things (IoT) devices. On the one hand, data collected from sensors will inevitably be affected or damaged by unpredictability. On the other hand, extending the active time of sensor networks has urgently aspired environmental monitoring. Using neural networks to design a data imputation algorithm can take advantage of the prior information stored in the models. This paper proposes a preprocessing algorithm to extract a subset for training a neural network on an IoT dataset, including time window determination, sensor aggregation, sensor exclusion and data frame shape selection. Moreover, we propose a data imputation algorithm using deep compressed sensing with generative models. It explores novel representation matrices and can impute data in the case of a high missing ratio situation. Finally, we test our subset extraction algorithm and data imputation algorithm on the EPFL SensorScope dataset, respectively, and they effectively improve the accuracy and robustness even with extreme data loss.

Determining nutrients, dry matter, and pH of liquid organic manures using visual and near-infrared spectrometry

In agriculture, overfertilization with liquid organic manures (LOM) is causing environmental issues including eutrophication of non-agricultural ecosystems and nitrate pollution of groundwater. To avoid such problems, a precise and demand-oriented fertilization with LOM is needed. This can only be achieved if the nutrient composition of the LOM is known. However, traditional chemical analysis is cost- and time-intensive and furthermore dependent on a representative sample. Optical spectrometry in the visible and near-infrared range could provide an efficient alternative, if a chemometric calibration assures sufficient accuracy. To improve chemometric calibration, this study investigated several spectral preprocessing and regression algorithms, and compared predictions based either on dry or wet weight concentration. In addition, the capability of low-cost spectrometers was evaluated by simulating low-resolution spectra with smaller wavelength ranges. The reflectance spectra of 391 pig manure, 155 cattle manure, and 89 biogas digestate samples were used to predict plant macronutrients (N, P, K, Mg, Ca, S), micronutrients (Mn, Fe, Cu, Zn, B), dry matter (DM) and pH. The experiments demonstrate the general aptness of optical spectrometry to accurately predict DM, pH and all nutrients except boron in pig, cattle, and digestate LOM, even with simulated low-cost spectrometers. Best results show r2 between 0.80 and 0.97, ratios of performance to interquartile distance (RPIQ) between 2.1 and 7.8, and mean absolute errors normalized by the median (nMAE) between 5 and 36 %. The regression methods PLSR, LASSO, and least angle regression predominantly performed best. The innovative preprocessing methods named simple ratios (SR) and normalized differences (ND) proved to be very useful algorithms, especially for N and P predictions, outperforming the accuracy of classical techniques in several cases. Concentrations on dry weight basis improved predictions of K, Mn, and pH.

Emergency vehicles classification for traffic signal system using optimized transfer DenseNet201 model

&lt;span&gt;As a result of the rapid growth of the world population, traffic signaling systems for monitoring and controlling the roads have turned to be an important issue facing humanity. To effectively overcome this problem, an accurate method for congestion reduction on the roads should be used which has a direct relation between the population and the cars’ usage. Various approaches derived from deep and transfer learning have been investigated in this context. This research implemented an optimized transfer learning approach for densely connected convolutional neural network (DenseNet201) models for multiple classifications (non-emergency cars, ambulance, police, and firefighter). Due to the non-availability of public datasets, a customized dataset has been created. This paper aims to improve the performance accuracy of vehicle classification using certain preprocessing algorithms on the input images and testing various optimization methods. The performance accuracy of the proposed model is evaluated using k-folds cross-validation 20:80 for the test and training, respectively. The metrics which are used for comparison with other deep learning techniques are based on exactness, recall, accuracy and F1-score. Test outcomes specify that the proposed transfer model-based optimization outperforms alternative deep learning algorithms regarding vehicle accuracy in classifying and reaches 98.6%.&lt;/span&gt;

Classifying Participant Standing and Sitting Postures Using Channel State Information

Recently, channel state information (CSI) has been identified as beneficial in a wide range of applications, ranging from human activity recognition (HAR) to patient monitoring. However, these focused studies have resulted in data that are limited in scope. In this paper, we investigate the use of CSI data obtained from an ESP32 microcontroller to identify participants from sitting and standing postures in a many-to-one classification. The test is carried out in a controlled isolated environment to establish whether a pre-trained model can distinguish between participants. A total of 15 participants were recruited and asked to sit and stand between the transmitter (Tx) and the receiver (Rx), while their CSI data were recorded. Various pre-processing algorithms and techniques have been incorporated and tested on different classification algorithms, which have gone through parameter selection to enable a consistent testing template. Performance metrics such as the confusion matrix, accuracy, and elapsed time were captured. After extensive evaluation and testing of different classification models, it has been established that the hybrid LSTM-1DCNN model has an average accuracy of 84.29% and 74.13% for sitting and standing postures, respectively, in our dataset. The models were compared with the BedroomPi dataset and it was found that LSTM-1DCNN was the best model in terms of performance. It is also the most efficient model with respect to the time elapsed to sit and stand.

Recent Advancement in Prediction and Analyzation of Brain Tumour using the Artificial Intelligence Method

Brain tumour identification and categorization are critical for diagnosis and treatment. This work uses preprocessing and classification algorithms to discover and categories brain tumours. Gaussian smoothing reduces noise and improves image quality, Genetic Algorithms select and optimize features, Deep Learning-Based Segmentation accurately segments tumours, Local Binary Patterns (LBP) extract texture features, and K-Nearest Neighbors classify tumours. Gaussian smoothing reduces noise and improves brain imaging data. Genetic Algorithms extract the most relevant and discriminative features from preprocessed photos. A Deep Learning-Based Segmentation model accurately segments brain tumour regions using these features. After segmentation, Local Binary Patterns (LBP) extract tumour texture features. The K-Nearest Neighbors (KNN) method classifies tumours using these texture features to capture tumour spatial patterns. Our suggested brain tumour detection and classification method combines several techniques to increase accuracy and reliability. Gaussian smoothing and LBP feature extraction improve feature discrimination. Deep Learning-Based Segmentation and the KNN classifier ensure precise tumour location and robust classification. The proposed method will be tested on brain scans including tumour areas. Classification performance measures include accuracy, sensitivity, specificity, and AUC. This work will improve brain tumour detection and classification methods for more accurate diagnoses and treatment planning. The primary goal of the work is to enhance brain tumour identification and categorization using pre processing, classification algorithms, and advanced techniques, ultimately improving diagnosis and treatment outcomes.

An Adaptive Fuzzy C-Means segmentation and deep learning model for efficient mammogram classification using VGG-Net

Breast cancer mortality can be prevented by only early, accurate mammography screening and diagnosis. Although CNN-based computer-aided diagnosis (CAD) systems for breast cancer have made tremendous progress recently, accurate identification of mammography lesions is still difficult because of poor signal-to-noise ratio (SNR) and physiological features. In this manuscript, an Adaptive Fuzzy C-Means Segmentation and Deep Learning Model for Efficient Mammogram Classification Using VGG-Net (AFCM-DCNN) is proposed. The input image is given to Grey Code Approximation Pre-processing (GCAP) algorithm to enhance the quality of image by adjusting the pixel contrast. The preprocessed image is given to Adaptive Fuzzy C-Means (AFCM) algorithm and is applied in segmenting dominant regions in an input image. But in conventional AFCM technique, the centroid values get generated randomly, which consumes more computational time. Hence to enhance the performance of traditional AFCM, centroid value is optimally chosen by means of optimization algorithm. A technique for classifying images called DCNN analyses the input image and categorizes it as either benign, malignant and normal. The method extracts the features of the image and train with VGG-16 Net classifier. The neurons at the output layer have been designed to compute Class Centric Disease Support (CCDS) towards various classes. Accordingly, the mammogram class is identified towards detecting the brain tumor. The performance of the proposed method AFCM-DCNN exhibits higher accuracy of 29.3%, 25.6% and 24.6%, higher sensitivity of 15.4%, 16.6% compared with the existing methods. Therefore, in future work, hope to enhance the performance depending on transfer learning with similar data.

3D Pre-Processing Algorithm for MRI Images of Different Stages of AD

Alzheimer’s disease (AD) is a degenerative neuronal brain disorder resulting in memory loss, skills, and cognitive changes. The disorder’s primary diagnostic tests are defined as total brain atrophy and hippocampal atrophy. Early diagnosis is significant, and automatic systems design is necessary for this disorder. Potential biomarkers for AD are described using a hippocampal magnetic resonance imaging volumetry system with certain limitations. For the definite identification of the hippocampus region, pre-processing of the 3D MRI images of AD is necessary. The filtering and histogram-based pre-processing techniques enhance the region of interest, which helps in effectively segmenting the biomarker, the hippocampus. The median and eight histogram clippings are defined to be 98% efficient pre-processing techniques with the comparison of image quality parameters and statistical analysis. Thus an algorithm for pre-processing of the 3D MRI images of stages of AD is designed for the further process of identification.

Computer Aided Classification of X-ray Images from Pediatric Pneumonia Subjects Collected in Developing Countries

Pneumonia is a lower tract respiratory infection due to bacteria or viruses. It is a severe disease in the pediatric population. Pneumonia is the leading cause of mortality in children under five years worldwide. One of the problems with pneumonia is the diagnosis, as the symptoms of pneumonia may overlap with other diseases, such as asthma and bronchiolitis. In this work, we propose to develop a method for classifying pneumonia and non-pneumonia using X-ray images. We collected 60 X-ray images from Dr. Sardjito Hospital, Yogyakarta, Indonesia, and the dataset from Kaggle. We processed these images through pre-processing algorithms to enhance the image quality, segmentation, white pixel computation, and classification. The novelty of our method is using the ratio of the white pixels from edge detection using the Canny algorithm with the white pixels from segmentation for classifying pneumonia/non-pneumonia. In the Kaggle dataset, our proposed method achieved an accuracy of 86.7%, a sensitivity of 100%, and a specificity of 85%. The classification using the dataset from Dr. Sardjito Hospital yields sensitivity, specificity, and accuracy of 80%, 60%, and 66.7%, respectively. Despite the low performance in the results, we proved our novel feature, ratio of white pixels, can be used to classify pneumonia/non-pneumonia. We also identified that the local dataset is essential in the algorithm development as it has a different quality from the dataset from modern countries. Further, our simple method can be developed further to support pneumonia diagnosis in resource-limited settings where the advanced computing devices or cloud connection are not available.