Batch Processing Method Research Articles

Distributed Batch Learning of Growing Neural Gas for Quick and Efficient Clustering

Growing neural gas (GNG) has been widely used in topological mapping, clustering and unsupervised tasks. It starts from two random nodes and grows until it forms a topological network covering all data. The time required for growth depends on the total amount of data and the current network nodes. To accelerate growth, we introduce a novel distributed batch processing method to extract the rough distribution called Distributed Batch Learning Growing Neural Gas (DBL-GNG). First, instead of using a for loop in standard GNG, we adopt a batch learning approach to accelerate learning. To do this, we replace most of the standard equations with matrix calculations. Next, instead of starting with two random nodes, we start with multiple nodes in different distribution areas. Furthermore, we also propose to add multiple nodes to the network instead of adding them one by one. Finally, we introduce an edge cutting method to reduce unimportant links between nodes to obtain a better cluster network. We demonstrate DBL-GNG on multiple benchmark datasets. From the results, DBL-GNG performs faster than other GNG methods by at least 10 times. We also demonstrate the scalability of DBL-GNG by implementing a multi-scale batch learning process in it, named MS-DBL-GNG, which successfully obtains fast convergence results. In addition, we also demonstrate the dynamic data adaptation of DBL-GNG to 3D point cloud data. It is capable of processing and mapping topological nodes on point cloud objects in real time.

Integrating Image Analysis and Machine Learning for Moisture Prediction and Appearance Quality Evaluation: A Case Study of Kiwifruit Drying Pretreatment.

The appearance of dried fruit clearly influences the consumer's perception of the quality of the product but is a subtle and nuanced characteristic that is difficult to quantitatively measure, especially online. This paper describes a method that combines several simple strategies to assess a suitable surrogate for the elusive quality using imaging, combined with multivariate statistics and machine learning. With such a convenient tool, this study also shows how one can vary the pretreatments and drying conditions to optimize the resultant product quality. Specifically, an image batch processing method was developed to extract color (hue, saturation, and value) and morphological (area, perimeter, and compactness) features. The accuracy of this method was verified using data from a case study experiment on the pretreatment of hot-air-dried kiwifruit slices. Based on the extracted image features, partial least squares and random forest models were developed to satisfactorily predict the moisture ratio (MR) during drying process. The MR of kiwifruit slices during drying could be accurately predicted from changes in appearance without using any weighing device. This study also explored determining the optimal drying strategy based on appearance quality using principal component analysis. Optimal drying was achieved at 60 °C with 4 mm thick slices under ultrasonic pretreatment. For the 70 °C, 6 mm sample groups, citric acid showed decent performance.

The Effectiveness of Carbonized Neem Leaves Adsorbent for the Removal of Phenol from Synthesized Effluent Using the Batch Process Method

The adsorption of phenol in aqueous solution was investigated using carbonized neem leaves adsorbents. The effect of various factors such as contact time, phenol concentration, pH, temperature, and amount of adsorbent on the adsorption capacity of the adsorbents was determined. The results show that maximum adsorption was obtained at a contact time of 150 minutes, at a phenol concentration of 50 mg/L, optimum adsorption temperature of 45⁰C, pH of 2 and adsorbent dosage of 3 g/L. The equilibrium data was well described by the Freundlich isotherm equation (R2= 0.955 ). This shows a heterogenous adsorption process. The kinetics of the process were explained by the intra-particle diffusion models (R2>0.9). This fitted data indicates a degree of boundary layer control and is not limited to intra-particle diffusion.

A Selective Migration-Based Improved GPR Modeling Method for Batch Process

A Selective Migration-Based Improved GPR Modeling Method for Batch Process

MBPPE: A Modular Batch Processing Platform for Electroencephalography

To ensure the accuracy and reliability of subsequent analysis, research on electroencephalogram (EEG) signals typically requires preliminary processing of large datasets to eliminate noise and artifacts. Traditional batch processing methods require substantial hardware resources while lacking flexible automated workflows and user-friendly interactions. To address these challenges, we have implemented a modular batch processing platform for EEG (MBPPE) that offers both local execution and private deployment options to meet the demands of efficient signal processing from individuals to laboratories. We modularize the processing methods and organize them into pluggable multi-task batch processes, providing asynchronous processing solutions. In addition, we extend user functions by introducing plugins and promoting collaborative interaction through data sharing, access control, and comment communication. Simultaneously, interactive features are integrated into the visualization design, enabling users to process and analyze data more intuitively and naturally. Currently, the platform integrates several commonly used data preprocessing and analysis techniques, providing a novel solution for batch processing of EEG signals.

Data-driven two-dimensional integrated control for nonlinear batch processes

Two-dimensional control has been considered as an effective strategy to accomplish high-accuracy tracking for batch processes because of its excellent learning ability and time-domain stability. However, being a model-based control method, the performance of the two-dimensional control system will inevitably decrease due to unknown uncertainties or unmodeled dynamics. In addition, the high computational cost and complex design process of the control system severely limit its application in batch processes. For this reason, this paper proposes a new data-driven two-dimensional integrated control (DDTDIC) method for nonlinear batch processes. In the presented control scheme, the P-type iterative learning control (ILC) is adopted along the batch-axis to ensure the convergence of the system, and the proportional-integral-differential (PID) control is used in the time-axis to reject the influence of real-time disturbance. The parameters of the PID controller are obtained by utilizing the virtual reference feedback tuning (VRFT) method. The entire design process of the control system only requires the input and output (I/O) data of the batch processes and does not depend on any explicit model information. The simulation results show that compared with the ILC and the two-dimensional control, the presented control method not only has faster convergence speed and smaller tracking error, but also the computational efficiency is improved by more than 40% and 50% respectively.

Cross-coupling indirect iterative learning control method for batch processes with time-varying uncertainties

For batch time-varying processeswith non-repetitive disturbances, a cross-coupling indirect iterative learning control (CC-iILC) is proposed. The set trajectory of the system on the single axis is accurately tracked by an indirect iterative learning control strategy with a PI controller for its time direction and a closed-loop feedback control strategy for the batch direction. Under the asymptotically stable condition of the two-dimensional (2D) dynamic model, the optimal control law is obtained by optimizing the H-infinity control function. To avoid the contour error during coupling, the cross-coupling technique is used to distribute the contour error to each axis for compensation. The stability condition of the indirect iterative learning controller is analyzed by a two-dimensional Fornasini-Marchesini (FM) batch dynamics model and two-dimensional robust H-infinity control theory. The feasibility and superiority of the proposed control method are verified by numerical simulation and experimental tests.

A Hybrid Missing Data Imputation Method for Batch Process Monitoring Dataset.

Batch process monitoring datasets usually contain missing data, which decreases the performance of data-driven modeling for fault identification and optimal control. Many methods have been proposed to impute missing data; however, they do not fulfill the need for data quality, especially in sensor datasets with different types of missing data. We propose a hybrid missing data imputation method for batch process monitoring datasets with multi-type missing data. In this method, the missing data is first classified into five categories based on the continuous missing duration and the number of variables missing simultaneously. Then, different categories of missing data are step-by-step imputed considering their unique characteristics. A combination of three single-dimensional interpolation models is employed to impute transient isolated missing values. An iterative imputation based on a multivariate regression model is designed for imputing long-term missing variables, and a combination model based on single-dimensional interpolation and multivariate regression is proposed for imputing short-term missing variables. The Long Short-Term Memory (LSTM) model is utilized to impute both short-term and long-term missing samples. Finally, a series of experiments for different categories of missing data were conducted based on a real-world batch process monitoring dataset. The results demonstrate that the proposed method achieves higher imputation accuracy than other comparative methods.

A Decision-Focused Learning Framework for Vessel Selection Problem

Maritime transportation safety is pivotal in international trade, with port state control (PSC) inspections being crucial to vessel safety. However, port authorities need to identify substandard vessels effectively because of resource constraints and high costs. Therefore, we propose robust predictive models and optimization strategies for vessel selection, using the random forest (RF) algorithm. We first use a traditional RF model serving as a benchmark, denoted as model M0. Then, we construct model M1 by refining the RF algorithm with a batch-processing method, thereby providing a better measure of the relative relationship between the predicted deficiency counts within a batch of ships. Then, we propose model M2, incorporating a decision-focused learning (DFL) framework into the tree construction process, enhancing the decision performance of the algorithm. In addition, we propose a variant model of M2, denoted as M2-0, considering the worst-case scenario when designing the decision loss function. By conducting experiments with data from the port of Hong Kong, we demonstrate that models M1 and M2 offer superior decision-making performance compared to model M0, and model M2 outperforms model M2-0 in both decision performance and stability. We further verify the robustness of these models by testing them under various instance scales. Overall, our study enhances the PSC inspection efficiency, ultimately bolstering maritime transportation safety.

Research on the digital transformation path of banks in the era of big data

Abstract This paper proposes a weighted large-scale data subspace clustering algorithm to enable it to adapt to the mega-customer environment for financial banks to respond quickly to customer data. Firstly, based on the K-means combined with a genetic algorithm, an improved method for the sensitivity problem of initial clustering center selection of K-means algorithm is proposed. By weighting the variables and streaming data batch processing method as a guide, the improvement method is proposed for the problem that the mean algorithm cannot identify the correct clustering center caused by the ultra-large-scale data environment, leading to the iteration number approaching infinity. The accuracy of the K-mean algorithm, the optimized initial clustering center algorithm, and the algorithm in this paper are 89.61%, 94.37% and 96.94%, respectively. In terms of running time, the highest running time of this algorithm is 10.96 seconds, which is faster than the running time of the other two algorithms. Finally, the financial analysis of the financial bank that completed the digital transformation with the help of the algorithm in this paper, the bank achieved a business of 150.832 billion yuan in 2021, an increase of 11% compared with the end of last year. Net profit achieved 44.883 billion yuan, an increase of 25.8% compared to the end of last year. Therefore, the algorithm in this paper has high advantages in terms of accuracy, efficiency, and practicality, proving that digital transformation can improve bank profits. It also provides a path and direction of transformation for various urban and agricultural commercial banks and other small credit unions.

The Research on Performance Characteristics of the Sliding Relief Valve

Based on the software AMESim, this paper introduced the operating principle and established the system models of relief valves. The effects on dynamic characteristics under the different parameters, including the number of overflow orifices, overflow orifice diameter and spring stiffness, were analyzed by the AMESim batch processing method. This paper compared and analyzed the curves and summarized the parameters which affect the performance characteristics of the relief valves. The result helps optimize the structure parameters and improve the performance characteristics of the relief valves.

Iterative Learning-Based Predictive Control Method for Electronic Grade Silicon Single Crystal Batch Process

The growth process of silicon single crystal (SSC) is a typical batch production process, which has the characteristics of batch size, variety, complex process, and intensive technology. In order to accurately control the diameter and thermal field temperature of the Czochralski (Cz) SSC batch production process to ensure that the actual production requirements of electronic-grade SSC are met, this paper proposes an iterative learning-based batch process predictive control strategy Model Predictive Control -Iterative Learning Control -Extended State Observer (MPC-ILC -ESO). The control strategy consists of two parts. The time axis is a dual Model Predictive Control (MPC) controller, which is mainly used to deal with disturbance suppression and constraints in the production process of a single batch of SSC. The iterative axis is the Extended State Observer (ESO) based Iterative Learning Control (ILC) control algorithm to deal with uncertainty and disturbance estimation and compensation in the multi-batch production process. The control effects of the time axis and the batch axis are superimposed as the final controlled object input to ensure the crystal diameter and stable thermal field temperature. Finally, the effectiveness of the proposed control strategy is verified based on the data analysis of the semiconductor industry production process.

Protection Schemes for DDoS, ARP Spoofing, and IP Fragmentation Attacks in Smart Factory

Industry Revolution 4.0 connects the Internet of Things (IoT) resource-constrained devices to Smart Factory solutions and delivers insights. As a result, a complex and dynamic network with a vulnerability inherited from the Internet becomes an attractive target for hackers to attack critical infrastructures. Therefore, this paper selects three potential attacks with the evaluation of the protections, namely (1) distributed denial of service (DDoS), (2) address resolution protocol (ARP) spoofing, and (3) Internet protocol (IP) fragmentation attacks. In the DDoS protection, the F1-score, accuracy, precision, and recall of the four-feature random forest with principal component analysis (RFPCA) model are 95.65%, 97%, 97.06%, and 94.29%, respectively. In the ARP spoofing, a batch processing method adopts the entropy calculated in the 20 s window with sensitivity to network abnormalities detection of various ARP spoofing scenarios involving victims’ traffic. The detected attacker’s MAC address is inserted in the block list to filter malicious traffic. The proposed protection in the IP fragmentation attack is implementing one-time code (OTC) and timestamp fields in the packet header. The simulation shows that the method detected 160 fake fragments from attackers among 2040 fragments.

MOBILE U-NET V3 AND BILSTM: PREDICTING STOCK MARKET PRICES BASED ON DEEP LEARNING APPROACHES

The development of reliable stock market models has enabled investors to make better-informed decisions. Investors may be able to locate companies that offer the highest dividend yields and lower their investment risks by using a trading strategy. The degree to which stock prices are significantly correlated, however, makes stock market analysis more complicated when using batch processing methods. The stock market prediction has entered a time of advanced technology with the rise of technological wonders like global digitalization. The significance of artificial intelligence models has greatly increased as a result of the significantly enhance in market capitalization. Because it builds a strong time-series framework based on Deep Learning (DL) for predicting future stock prices, the proposed study is novel. Deep learning has recently enjoyed considerable success in some domains due to its exceptional capacity for handling data. For instance, it is commonly used in financial disciplines such as trade execution strategies, portfolio optimization, and stock market forecasting. In this research, we propose a structure based on Mobile U-Net V3 and a hybrid of a (Mobile U-Net V3-BiLSTM) with BiLSTM to forecast the closing prices of Apple, Inc. and S&P 500 stock data. The Root Mean Squared Error (RMSE), Mean Squared Error (MSE), Pearson's Correlation (R), and Normalization Root Mean Squared Error (NRMSE) metrics were utilized to calculate the outcomes of the DL stock prediction methods. The Mobile U-Net V3-BiLSTM model outperformed other techniques for forecasting stock market prices.

A New Explanation of loading Rate Effect on Rock Compressive Strength

Apart from several qualitative descriptions, such as the delay crack propagation and the difference in energy absorption, the involved mechanism of loading rate effect (rate-dependence) on unconfined compressive strength (UCS) of rock is not clear. According to the statistical analysis of the dominant frequency of acoustic emission (AE) waveforms, this work proposed a new explanation of rate-dependence on rock UCS from a micro failure perspective. The uniaxial compression tests, coupling with AE signals monitoring, of five types of hard rocks under different loading rates were conducted. The rate-dependence on compressive strength of various rocks was investigated comprehensively. By using batch processing and Fast Fourier Transform methods, the dominant frequencies of AE waveforms associated with rock failures were extracted and analyzed to explain the rate-dependence of rock UCS. The results show that, as the loading rates increase, the UCS of most of the rocks increase, while the AE waveforms with low dominant frequency released during the entire loading process significantly decrease. Moreover, the percentage of AE waveforms with low dominant frequency has a negative correlation with the rock UCS. The micro-tensile failures can be characterized by AE waveforms with low dominant frequency and its reduction can be responsible for the increase in the UCS with increasing loading rates. The proposed explanation provides a new and reasonable perspective on the rate-dependence of rock UCS based on its intrinsic property.

Electrical Tree Image Segmentation Using Hybrid Multi Scale Line Tracking燗lgorithm

Electrical trees are an aging mechanism most associated with partial discharge (PD) activities in crosslinked polyethylene (XLPE) insulation of high-voltage (HV) cables. Characterization of electrical tree structures gained considerable attention from researchers since a deep understanding of the tree morphology is required to develop new insulation material. Two-dimensional (2D) optical microscopy is primarily used to examine tree structures and propagation shapes with image segmentation methods. However, since electrical trees can emerge in different shapes such as bush-type or branch-type, treeing images are complicated to segment due to manifestation of convoluted tree branches, leading to a high misclassification rate during segmentation. Therefore, this study proposed a new method for segmenting 2D electrical tree images based on the multi-scale line tracking algorithm (MSLTA) by integrating batch processing method. The proposed method, h-MSLTA aims to provide accurate segmentation of electrical tree images obtained over a period of tree propagation observation under optical microscopy. The initial phase involves XLPE sample preparation and treeing image acquisition under real-time microscopy observation. The treeing images are then sampled and binarized in pre-processing. In the next phase, segmentation of tree structures is performed using the h-MSLTA by utilizing batch processing in multiple instances of treeing duration. Finally, the comparative investigation has been conducted using standard performance assessment metrics, including accuracy, sensitivity, specificity, Dice coefficient and Matthew’s correlation coefficient (MCC). Based on segmentation performance evaluation against several established segmentation methods, h-MSLTA achieved better results of 95.43% accuracy, 97.28% specificity, 69.43% sensitivity rate with 23.38% and 24.16% average improvement in Dice coefficient and MCC score respectively over the original algorithm. In addition, h-MSLTA produced accurate measurement results of global tree parameters of length and width in comparison with the ground truth image. These results indicated that the proposed method had a solid performance in terms of segmenting electrical tree branches in 2D treeing images compared to other established techniques.

Modernization of Legacy Batch Processes: An Upgrade to Prolonged Batch Runs with Advanced Computing and Parallel Processing Techniques

In an era dominated by the need for speed and efficiency, long-running batch jobs in legacy systems significantly hinder organizational performance and scalability. This article explores the transformation of these batch jobs through the use of advanced computing and parallel processing techniques. We delve into the challenges posed by outdated batch processing methods and present solutions that leverage modern technology to enhance execution speed, reduce resource consumption, and improve overall system responsiveness.

Surface urban heat island effect and its driving factors for all the cities in China: Based on a new batch processing method

The urban heat island (UHI) effect has become a hot topic worldwide because of the double stress of global climate change and rapid urbanization. However, the deficiencies of existing methods hinder the monitoring and assessment of the surface urban heat island (SUHI) effect at the regional scale. This paper presented a new method to assess SUHI intensity, and then applied it in China. The results indicated that: (1) compared with other methods, this new method has some merits like higher accuracy, fewer operational steps, and higher portability. It enables accurate, semi-automated assessment of SUHI intensity at the regional scale. (2) SUHI effect exists in 81.02% of the cities in China. These cities were mainly concentrated in southeastern China, which presented the significant characteristic of spatial agglomeration. (3) Urban-rural differences in relative humidity, vegetation coverage, evapotranspiration, waterbody coverage, and the number of industrial enterprises are the main driving factors for the SUHI effect in China. Furthermore, there are significant interactions among these driving factors. (4) The spatial heterogeneity of the influence directions and intensities of many driving factors is obvious. Therefore, traditional regression models should be utilized more cautiously because they cannot capture the spatial heterogeneity of variables. (5) The hypothesis of “the larger the city size, the higher the SUHI intensity” is not valid, so more attention should be paid to small cities in the future.

Data-Driven Fault Detection in Industrial Batch Processes Based on a Stochastic Hybrid Process Model

This paper presents a novel fault detection approach for industrial batch processes. The batch processes under consideration are characterized by the interaction between discrete system modes and non-stationary continuous dynamics. Therefore, a stochastic hybrid process model (SHPM) is introduced, where process variables are modeled as time-variant Gaussian distributions, which depend on hidden system modes. Transitions between the system modes are assumed to be either autonomous or to be triggered by observable events such as on/off signals. The model parameters are determined from training data using expectation-maximization techniques. A new fault detection algorithm is proposed, which assesses the likelihoods of sensor signals on the basis of the stochastic hybrid process model. Evaluation of the proposed fault detection system has been conducted for a penicillin production process, with the results showing a significant improvement over the existing baseline methods.Note to Practitioners—Automatic fault detection makes it possible to limit the effects of faults by taking countermeasures at an early stage. In this work, a data-driven fault detection method for industrial batch processes is proposed, in which the underlying process model is learned from training data. The proposed fault detection system can be used for various industrial batch processes without the need for complex and error-prone manual configuration. In contrast to many other data-driven approaches such as neural networks, only a few process cycles are required to create a robust process model. It should be noted that in data-driven fault detection methods, the training data should cover a large part of the process states that occur during error-free process cycles. The developed method is therefore particularly suitable for cyclical processes, which, however, can have alternative process paths and variability between the process cycles.

Algorithmic Trading Stock Price Model

Abstract: Stock market price prediction is a difficult undertaking that generally requires a lot of human-computer interaction. Traditional batch processing methods cannot be used effectively for stock market analysis due to the linked nature of stock prices. This project presents an online learning technique that employs a recurrent neural network of some sort (RNN) called Long Short-Term Memory (LSTM), which uses stochastic gradient descent to update the weights for individual data points. When compared to existing stock price prediction systems, this will yield more accurate results. With varying sizes of data, the network is trained and evaluated for accuracy, and the results are tallied. A comparison with respect to accuracy is then performed against an Artificial Neural Network. Traditional approaches to securities market analysis and stock value prediction embrace basic analysis, that appearance at a stock's past performance and therefore the general believability of the corporate itself, and applied mathematics analysis, that is exclusively involved with computation and distinguishing patterns available value variation.