Training Data Sequence Research Articles

Advancements in intrusion detection: A lightweight hybrid RNN-RF model.

Computer networks face vulnerability to numerous attacks, which pose significant threats to our data security and the freedom of communication. This paper introduces a novel intrusion detection technique that diverges from traditional methods by leveraging Recurrent Neural Networks (RNNs) for both data preprocessing and feature extraction. The proposed process is based on the following steps: (1) training the data using RNNs, (2) extracting features from their hidden layers, and (3) applying various classification algorithms. This methodology offers significant advantages and greatly differs from existing intrusion detection practices. The effectiveness of our method is demonstrated through trials on the Network Security Laboratory (NSL) and Canadian Institute for Cybersecurity (CIC) 2017 datasets, where the application of RNNs for intrusion detection shows substantial practical implications. Specifically, we achieved accuracy scores of 99.6% with Decision Tree, Random Forest, and CatBoost classifiers on the NSL dataset, and 99.8% and 99.9%, respectively, on the CIC 2017 dataset. By reversing the conventional sequence of training data with RNNs and then extracting features before applying classification algorithms, our approach provides a major shift in intrusion detection methodologies. This modification in the pipeline underscores the benefits of utilizing RNNs for feature extraction and data preprocessing, meeting the critical need to safeguard data security and communication freedom against ever-evolving network threats.

Scaling Law for Recommendation Models: Towards General-Purpose User Representations

Recent advancement of large-scale pretrained models such as BERT, GPT-3, CLIP, and Gopher, has shown astonishing achievements across various task domains. Unlike vision recognition and language models, studies on general-purpose user representation at scale still remain underexplored. Here we explore the possibility of general-purpose user representation learning by training a universal user encoder at large scales. We demonstrate that the scaling law is present in user representation learning areas, where the training error scales as a power-law with the amount of computation. Our Contrastive Learning User Encoder (CLUE), optimizes task-agnostic objectives, and the resulting user embeddings stretch our expectation of what is possible to do in various downstream tasks. CLUE also shows great transferability to other domains and companies, as performances on an online experiment shows significant improvements in Click-Through-Rate (CTR). Furthermore, we also investigate how the model performance is influenced by the scale factors, such as training data size, model capacity, sequence length, and batch size. Finally, we discuss the broader impacts of CLUE in general.

A Learning-based Methodology for Scenario-aware Mapping of Soft Real-time Applications onto Heterogeneous MPSoCs

Soft real-time streaming applications often process input data that evoke varying workloads for their tasks. This may lead to high energy consumption or deadline misses in case their mapping onto a heterogeneous MPSoC target architecture is not adapted, e.g., when tasks with high execution times for the current input are assigned to resources of low computational power. To handle the vast variety of different input data, we propose to cluster data with similar execution characteristics into so-called data scenarios for which we determine specialized mappings by performing a scenario-aware design space exploration (DSE). A runtime manager (RTM) uses these mappings to adapt the execution of the running applications to their upcoming input by first identifying their best-suited scenarios. Subsequently, the RTM selects mappings considering their identified scenarios, which minimize the total number of deadline misses and the consumed energy. We embed the RTM into hybrid application mapping (HAM); ergo, performing time-consuming optimizations offline. In this article, we propose a novel data-scenario-aware HAM methodology that can cope with multiple applications and comprises two novel scenario-based mapping selection algorithms: Inter-Application Resource Mediation Mapping introduces barely any runtime overhead. Adaptive multi-app mapping selection is highly adaptive to changes in the application workload but imposes a small runtime overhead. Our HAM approach is fully automated and uses machine-learning techniques to learn the selection of suitable mappings from training data sequences at design time. Experiments on three differently complex target architectures show that our proposed approach consistently outperforms existing state-of-the-art solutions regarding the number of deadline misses and consumed energy.

A Pattern Dictionary Method for Anomaly Detection.

In this paper, we propose a compression-based anomaly detection method for time series and sequence data using a pattern dictionary. The proposed method is capable of learning complex patterns in a training data sequence, using these learned patterns to detect potentially anomalous patterns in a test data sequence. The proposed pattern dictionary method uses a measure of complexity of the test sequence as an anomaly score that can be used to perform stand-alone anomaly detection. We also show that when combined with a universal source coder, the proposed pattern dictionary yields a powerful atypicality detector that is equally applicable to anomaly detection. The pattern dictionary-based atypicality detector uses an anomaly score defined as the difference between the complexity of the test sequence data encoded by the trained pattern dictionary (typical) encoder and the universal (atypical) encoder, respectively. We consider two complexity measures: the number of parsed phrases in the sequence, and the length of the encoded sequence (codelength). Specializing to a particular type of universal encoder, the Tree-Structured Lempel–Ziv (LZ78), we obtain a novel non-asymptotic upper bound, in terms of the Lambert W function, on the number of distinct phrases resulting from the LZ78 parser. This non-asymptotic bound determines the range of anomaly score. As a concrete application, we illustrate the pattern dictionary framework for constructing a baseline of health against which anomalous deviations can be detected.

ISAIR: Deep inpainted semantic aware image representation for background subtraction

Nowadays, deep learning is impacting computer vision applications significantly by learning to extract and describe the essential features that can assist in solving a specific problem. One of the main drawbacks of deep learning-based approaches is their requirement of annotated training data. For instance, in the area of background subtraction, a sequence of annotated training data that demonstrates the foreground and background of the scene is required; such a sequence is required from different scenarios the module performs. Hence, such methods cannot perform directly in new environments, and they expect training data from the new scene. In order to benefit from the high generalization ability and easy-to-use capability of traditional background subtraction techniques, and rich feature representation of deep learning approaches, this paper utilized deep learning based techniques and investigated the best image representation that can assist conventional background subtraction approaches. The proposed image representation is based on the detection of objects which are semantically moving and inpainting the parts of the images related to such objects. Experiments and results show that the proposed image representation can have the online and portable capability of traditional techniques and improve their accuracy by approximately 20% when a conventional type of monocular camera is used.

Comparative Analysis of Machine Learning Models for Day-Ahead Photovoltaic Power Production Forecasting

A main challenge for integrating the intermittent photovoltaic (PV) power generation remains the accuracy of day-ahead forecasts and the establishment of robust performing methods. The purpose of this work is to address these technological challenges by evaluating the day-ahead PV production forecasting performance of different machine learning models under different supervised learning regimes and minimal input features. Specifically, the day-ahead forecasting capability of Bayesian neural network (BNN), support vector regression (SVR), and regression tree (RT) models was investigated by employing the same dataset for training and performance verification, thus enabling a valid comparison. The training regime analysis demonstrated that the performance of the investigated models was strongly dependent on the timeframe of the train set, training data sequence, and application of irradiance condition filters. Furthermore, accurate results were obtained utilizing only the measured power output and other calculated parameters for training. Consequently, useful information is provided for establishing a robust day-ahead forecasting methodology that utilizes calculated input parameters and an optimal supervised learning approach. Finally, the obtained results demonstrated that the optimally constructed BNN outperformed all other machine learning models achieving forecasting accuracies lower than 5%.

Parkinson’s Disease Classification using Various Advanced Neural Network Classifiers

This paper proposes the application of Online Meta-neuron Based Learning Algorithm (OMLA), Self adaptive Resource Allocation Network (SRAN) and Projection Based Learning Meta-cognitive Radial Basis Functional Network (PBL-McRBFN) for Parkinson’s disease classification. This is the first journal paper to apply the concept of OMLA, SRAN and PBL-McRBFN for Parkinson’s disease classification. Online Meta-neuron based Learning Algorithm (OMLA) is a newly evolved network applied for Parkinson’s disease classification. This classifier make use of both global and local information of the network. Self Adaptive Resource Allocation Network (SRAN) consists of self adaptive control parameters that changes training data sequence, develop network architecture and learns network parameters. Also, repeated learning samples are removed with the help of this algorithm, hence training time and over flow problems can be minimized. The Projection Based Learning algorithm determines the output parameters of the network such that the energy function is minimum. The result shows the comparison of efficiency for the three networks in classification of Parkinson’s disease.

The Impact of Training Data Sequence on the Performance of Neuro-Fuzzy Rainfall-Runoff Models with Online Learning

The learning algorithms in many of conventional Neuro-Fuzzy Systems (NFS) are based on batch or global learning where all parameters of the fuzzy system are optimized off-line. Although these models have frequently been used, they suffer from a reduced flexibility in their architecture as the number of rules need to be predefined by the user. This study uses a Dynamic Evolving Neural Fuzzy Inference System (DENFIS) in which an evolving, online clustering algorithm, the Evolving Clustering Method (ECM), is implemented. This study focused on evaluating the performance of this model in capturing the rainfall-runoff process and rainfall-water level relationship. The two selected study catchments are located in an urban tropical and in a semi-urbanized area, respectively. The first catchment, Sungai Kayu Ara (23.22 km2), is located in Malaysia, with 10-min rainfall-runoff time-series from which 30 major events are used. The second catchment, Dandenong (272 km2), is located in Victoria, Australia, with daily rainfall and river stage (water level) data from which 11 years of data is used. DENFIS results were then compared with two groups of benchmark models: a regression-based data-driven model known as the Autoregressive Model with Exogenous Inputs (ARX) for both study sites, and physical models Hydrologic Engineering Center–Hydrologic Modelling System (HEC–HMS) and Storm Water Management Model (SWMM) for Sungai Kayu Ara and Dandenong catchments, respectively. DENFIS significantly outperformed the ARX model in both study sites. Moreover, DENFIS was found comparable if not superior to HEC–HMS and SWMM in Sungai Kayu Ara and Dandenong catchments, respectively. A sensitivity analysis was then conducted on DENFIS to assess the impact of training data sequence on its performance. Results showed that starting the training with datasets that include high peaks can improve the model performance. Moreover, datasets with more contrasting values that cover wide range of low to high values can also improve the DENFIS model performance.

Attractor-based models for individual and groups’ forecasting

In this paper the questions of the attractors’ application in case of the development of the forecasting models on the base of the strictly binary trees have been considered. Usually, these models use the short time series as the training data sequence. The application of the principles of the attractors’ forming on the base of the long time series will allow creating the training data sequence more reasonably. The offered approach to creation of the training data sequence for the forecasting models on the base of the strictly binary trees was applied for the individual and groups’ forecasting of time series. At the same time the problems of one-objective and multiobjective optimization on the base of the modified clonal selection algorithm have been considered. The reviewed examples confirm the efficiency of the attractors’ application in sense of minimization of the used quality indicators of the forecasting models, and also the forecasting errors on 1 – 5 steps forward. Besides, the minimization of time expenditures for the development of the forecasting models is provided.

Attractors in the Development Problem of the Forecasting Model on the Base of the Strictly Binary Trees

In this paper the questions of the choice of the training data sequence for the forecasting model on the base of the strictly binary trees which is usually applied to time series with a short length of actual part (about 15 – 30 elements) have been considered. It is offered to use the principles of the attractors’ forming in the presence of the long time series that will allow creating the training data sequence more reasonably. The reviewed examples confirm the efficiency of the attractors’ use in sense of minimization of the affinity indicator of the forecasting model, and also the forecasting errors on 1 – 5 steps forward. Besides, the minimization of time expenditures on development of the forecasting model is provided.

Probabilistic ensemble Fuzzy ARTMAP optimization using hierarchical parallel genetic algorithms

In this study, a comprehensive methodology for overcoming the design problem of the Fuzzy ARTMAP neural network is proposed. The issues addressed are the sequence of training data for supervised learning and optimum parameter tuning for parameters such as baseline vigilance. A genetic algorithm search heuristic was chosen to solve this multi-objective optimization problem. To further augment the ARTMAP’s pattern classification ability, multiple ARTMAPs were optimized via genetic algorithm and assembled into a classifier ensemble. An optimal ensemble was realized by the inter-classifier diversity of its constituents. This was achieved by mitigating convergence in the genetic algorithms by employing a hierarchical parallel architecture. The best-performing classifiers were then combined in an ensemble, using probabilistic voting for decision combination. This study also integrated the disparate methods to operate within a single framework, which is the proposed novel method for creating an optimum classifier ensemble configuration with minimum user intervention. The methodology was benchmarked using popular data sets from UCI machine learning repository.

Heterogeneous Web Data Extraction Algorithm Based On Modified Hidden Conditional Random Fields

As it is of great importance to extract useful information from heterogeneous Web data, in this paper, we propose a novel heterogeneous Web data extraction algorithm using a modified hidden conditional random fields model. Considering the traditional linear chain based conditional random fields can not effectively solve the problem of complex and heterogeneous Web data extraction, we modify the standard hidden conditional random fields in three aspects, which are 1) Using the hidden Markov model to calculate the hidden variables, 2) Modifying the standard hidden conditional random fields through two stages. In the first stage, each training data sequence is learned using hidden Markov model, and then implicit variables can be visible. In the second stage, parameters can be learned for a given sequence. (3) The objective functions of hidden conditional random fields are revised, and the heterogeneous Web data are extracted by maximizing the posterior probability of the modified hidden conditional random fields. Finally, experiments are conducted to make performance evaluation on two standard datasets-EData dataset and Research Papers dataset. Compared with the existing Web data extraction methods, it can be seen that the proposed algorithm can extract useful information from heterogeneous Web data effectively and efficiently.

SIMO Channel Estimation Using Space-Time Signal Subspace Projection and Soft Information

We consider the channel estimation of a time-slotted wireless communication system with a mobile user and a base station, where the base station employs an M-element (M >; 1) antenna array. The uplink single-input multiple-output (SIMO) channel is usually estimated by training sequence within each time slot. To improve the estimation performance, the channel estimate is often refined by projecting it to the corresponding spatial signal subspace. However, this projection will not work when the number of resolvable multipath rays is larger than that of the antenna array elements, which makes the channel matrix full row rank. In this paper, we formulate the channel estimation under the space-time signal model for this full-row-rank case, and propose a new method by space-time signal subspace projection using both training and unknown data sequences. To further improve the accuracy of the channel estimate, the soft information fed back from the decoder can be used. By involving this soft information, we propose another new channel estimation method. This method approximately follows the maximum likelihood (ML) criterion and is therefore referred to as the approximated ML channel estimation. Numerical results show that these methods can be performed separately or jointly to improve the performance of channel estimation by training sequences.

A meta-cognitive learning algorithm for a Fully Complex-valued Relaxation Network

This paper presents a meta-cognitive learning algorithm for a single hidden layer complex-valued neural network called "Meta-cognitive Fully Complex-valued Relaxation Network (McFCRN)". McFCRN has two components: a cognitive component and a meta-cognitive component. A Fully Complex-valued Relaxation Network (FCRN) with a fully complex-valued Gaussian like activation function (sech) in the hidden layer and an exponential activation function in the output layer forms the cognitive component. The meta-cognitive component contains a self-regulatory learning mechanism which controls the learning ability of FCRN by deciding what-to-learn, when-to-learn and how-to-learn from a sequence of training data. The input parameters of cognitive components are chosen randomly and the output parameters are estimated by minimizing a logarithmic error function. The problem of explicit minimization of magnitude and phase errors in the logarithmic error function is converted to system of linear equations and output parameters of FCRN are computed analytically. McFCRN starts with zero hidden neuron and builds the number of neurons required to approximate the target function. The meta-cognitive component selects the best learning strategy for FCRN to acquire the knowledge from training data and also adapts the learning strategies to implement best human learning components. Performance studies on a function approximation and real-valued classification problems show that proposed McFCRN performs better than the existing results reported in the literature.

Improved GART Neural Network Model for Pattern Classification and Rule Extraction With Application to Power Systems

Generalized adaptive resonance theory (GART) is a neural network model that is capable of online learning and is effective in tackling pattern classification tasks. In this paper, we propose an improved GART model (IGART), and demonstrate its applicability to power systems. IGART enhances the dynamics of GART in several aspects, which include the use of the Laplacian likelihood function, a new vigilance function, a new match-tracking mechanism, an ordering algorithm for determining the sequence of training data, and a rule extraction capability to elicit if-then rules from the network. To assess the effectiveness of IGART and to compare its performances with those from other methods, three datasets that are related to power systems are employed. The experimental results demonstrate the usefulness of IGART with the rule extraction capability in undertaking classification problems in power systems engineering.

A sequential learning algorithm for self-adaptive resource allocation network classifier

This paper addresses sequential learning algorithm for self-adaptive resource allocation network classifier. Our approach makes use of self-adaptive error based control parameters to alter the training data sequence, evolve the network architecture, and learn the network parameters. In addition, the algorithm removes the training samples which are similar to the stored knowledge in the network. Thereby, it avoids the over-training problem and reduces the training time significantly. Use of misclassification information and hinge loss error in growing/learning criterion helps in approximating the decision function accurately. The performance evaluation using balanced and imbalanced data sets shows that the proposed algorithm generates minimal network with lesser computation time to achieve higher classification performance.

A novel Euclidean quality threshold ARTMAP network and its application to pattern classification

This paper introduces a novel neural network model known as the Euclidean quality threshold ARTMAP (EQTAM) network and its application to pattern classification. The model is constructed based on fuzzy ARTMAP (FAM) and the quality threshold clustering principle. The main objective of EQTAM is to overcome the effects of training data sequences on FAM and, at the same time, to improve its classification performance. Several artificial data sets and benchmark medical data sets are used to evaluate the effectiveness of the proposed model. Performance comparisons between EQTAM and ARTMAP-based as well as other classifiers are made. From the experimental results, it can be observed that EQTAM is able to produce good results. More importantly, the performance of EQTAM is robust against the effect of training data orders or sequences.

A stochastic analysis of an iterative semi-blind beamformer for TDMA systems

We describe an iterative semi-blind (SB) beamforming algorithm for time-division multiple-access (TDMA) systems and analyze its performance using a stochastic model. A least-squares approximation for constant modulus signals leads to a relatively simple and practical SB algorithm. The beamformer weights are calculated using a two-step procedure: initial weights are computed using training data and these are refined by the SB algorithm. The main goal of this paper is to analyze the beamforming algorithm using Wiener filter theory. In one approach, we develop a stochastic model using a Gaussian approximation that simplifies a fourth-order moment of the array data. We also provide an exact analysis and compare the results with that of the Gaussian approximation. Furthermore, we compare the proposed SB algorithm with the regularized semi-blind (RSB) technique using simulated and real TDMA data. The performance of the SB algorithm is also analyzed with short as well as long training data sequences in asynchronous interference scenarios. Experimental results show that the proposed SB beamforming algorithm suppresses cochannel interference affecting the coded data, and outperforms the RSB algorithm.

USING OF AN ADAPTIVE MATCHED FILTER IN SELF ADAPTIVE EQUALIZATION

This paper deals with a new proposal technique for improving the performance of a communication link. This is carried out by using an adaptive filter cascaded with the matched filter at the receiving end. When this proposal is used to equalizing a communication channel , no need to a training data sequence at the start of the system operation. So, it is important to use it in self adaptive equalization [1]. The matched filter is used to maximize signal to noise ratio at the receiving end. Since the matched filter is designed to match the transmitted signal, which is different from the received signal due to the channel effect the output of the matched filter is different from the designed one. An adaptive filter may be used to avoid these drawbacks caused by the channel since it has the ability to track the channel variation. The paper presents a system based on using adaptive filter together with a matched filter at the receiver. To check the performance of the proposal, a BARKER code is used to modulate four level data. At the receiver, the adaptive filter is cascaded with the matched filter and adapts its weights to get ideal auto-correlation function of the BARKER code. The basic idea in the adaptation process is based on the shape of the auto-correlation function, which has alternative zeroes. So, the adaptive filter adapts its output to zero at these instants and stop adaptation at the next instants, which solves the problem of the distortion in the matched filter output. The main advantage of the adaptation processor of the adaptive filter will modify the tap weights at double the baud interval i.e, slower processor is sufficient to be used. Computer simulation is carried out to test the performance of the proposal. It clarifies the improvement caused when the proposal is used.

Artificial neural networks for rapid WWTP performance evaluation: Methodology and case study

Reliable performance evaluation of wastewater treatment plants (WWTPs) can be done by simulating the plant behavior over a wide range of influent disturbances, including series of rain events with different intensity and duration, seasonal temperature variations, holiday effects, etc. Such simulation-based WWTP performance evaluations are in practice limited by the long simulation time of the mechanistic WWTP models. By moderate simplification (avoiding big losses in prediction accuracy) of the mechanistic WWTP model only a limited reduction of the simulation time can be achieved. The approach proposed in this paper combines an influent disturbance generator with a mechanistic WWTP model for generating a limited sequence of training data (4 months of dynamic data). An artificial neural network (ANN) is then trained on the available WWTP input-output data, and is subsequently used to simulate the remainder of the influent time series (20 years of dynamic data) generated with the influent disturbance generator. It is demonstrated that the ANN reduces simulation time by a factor of 36, even when including the time needed for the generation of training data and for ANN training. For repeated integrated urban wastewater system simulations that do not require repeated training of the ANN, the ANN reduces simulation time by a factor of 1300 compared to the mechanistic model. ANN prediction of effluent ammonium, BOD 5 and total suspended solids was good when compared to mechanistic WWTP model predictions, whereas prediction of effluent COD and total nitrogen concentrations was a bit less satisfactory. With correlation coefficients R 2 > 0.95 and prediction errors lower than 10%, the accuracy of the ANN is sufficient for applications in simulation-based WWTP design and simulation of integrated urban wastewater systems, especially when taking into account the uncertainties related to mechanistic WWTP modeling.