Multiple Individual Models Research Articles

Ejection of Real-Time Malicious Intrusion and Attacks in IoT Empowered Infrastructure

The project recognizes the pervasive threat of computer viruses, malware, and hostile attacks on computer networks, highlighting the critical role of intrusion detection as a proactive defense technology. The project introduces a novel approach based on deep learning to identify and mitigate cybersecurity vulnerabilities and breaches in IoT-driven cyber-physical systems, aiming for enhanced security measures. The project's objective is to elevate intrusion detection beyond the limitations of traditional systems by addressing issues like accuracy, detection effectiveness, and reducing false positives. This emphasizes the advancement and innovation in cybersecurity. To achieve the project's goals, the method employs a generative adversarial network, a cutting-edge deep learning technique. Additionally, it distinguishes itself by contrasting unsupervised and deep learning-based discriminative approaches, showcasing a comprehensive and effective approach to cybersecurity. In our project, we successfully implemented an ensemble method to boost predictive accuracy by integrating multiple individual models. Particularly noteworthy is the inclusion of a hybrid architecture, combining Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM), denoted as CNN+LSTM. This hybrid model achieved an impressive accuracy of 99% when applied to the KDD-Cup dataset, underscoring the efficacy of our ensemble technique for intrusion detection in IoT- based cybersecurity infrastructures.

Feature Bagging with Nested Rotations (FBNR) for anomaly detection in multivariate time series

Detecting anomalies in multivariate time series poses a significant challenge across various domains. The infrequent occurrence of anomalies in real-world data, as well as the lack of a large number of annotated samples, makes it a complex task for classification algorithms. Deep Neural Network approaches, based on Long Short-Term Memory (LSTMs), Autoencoders, and Variational Autoencoders (VAEs), among others, prove effective with handling imbalanced data. However, the same does not follow when such algorithms are applied on multivariate time-series, as their performance degrades significantly. Our main hypothesis is that the above is due to anomalies stemming from a small subset of the feature set. To mitigate the above issues in the multivariate setting, we propose forming an ensemble of base models by combining different feature selection and transformation techniques. The proposed processing pipeline includes applying a Feature Bagging techniques on multiple individual models, which considers separate feature subsets for each specific model. These subsets are then partitioned and transformed using multiple nested rotations derived from Principal Component Analysis (PCA). This approach aims to identify anomalies that arise from only a small portion of the feature set while also introduces diversity by transforming the subspaces. Each model provides an anomaly score, which are then aggregated, via an unsupervised decision fusion model. A semi-supervised fusion model was also explored, in which a Logistic Regressor was applied on the individual model outputs. The proposed methodology is evaluated on the Skoltech Anomaly Benchmark (SKAB), containing multivariate time series related to water flow in a closed circuit, as well as the Server Machine Dataset (SMD), which was collected from a large Internet company. The experimental results reveal that the proposed ensemble technique surpasses state-of-the-art algorithms. The unsupervised approach demonstrated a performance improvement of 2% for SKAB and 3% for SMD, compared to the baseline models. In the semi-supervised approach, the proposed method achieved a minimum of 10% improvement in terms of anomaly detection accuracy.

Machine learning chain models for multi-response prediction of electrical equipment in substation subjected to earthquakes

Engineering structures often exhibit multiple potential vulnerable positions during strong earthquakes, such as porcelain insulators and connection flanges of electrical equipment in substations. This paper proposes two types of machine learning (ML) chain models, where multiple individual models are linked end to end, to predict multiple peak seismic responses of substation equipment at the same time using intensity measures (IMs). One is a simple chain where the next input is the previous output, while in another one, the next input is combined by IMs and the previous output. The training mechanisms of ML chain models are presented by means of bio-inspired multi-objective optimization techniques for the hyperparameters of multiple individual ML models. A case study on a 1100 kV transformer bushing is then conducted. Artificial neural network-gradient boosting regression and artificial neural network-kernel ridge regression chain models are respectively established for predicting the peak stresses at two most vulnerable positions. Both the evaluation indicators and shaking table tests show enough prediction accuracy of the two ML chain models. Such chain models are qualified for identifying the initial damage to equipment, which can be employed for assisting post-earthquake rapid judgement and relief.

Mitigating insider’s threats using support vector machine and k-nearest Neighbour

Addressing insider’s threats is a critical challenge in organizational security. This study presents the development and evaluation of a hybrid machine learning model aimed at enhancing insider’s threat detection effectiveness. The escalating risks associated with insider’s threats necessitated advance detection mechanisms to mitigate potential breaches. Leveraging the strengths of multiple individual models, including Support Vector Machine (SVM) and K-nearest Neighbour (KNN), the hybrid model addressed this challenge by improving detection accuracy while minimizing false positives. Through rigorous evaluation, the hybrid model demonstrates remarkable performance, achieving an accuracy of 99%, with precision, recall, and F1 score of 99%, 98%, and 97% respectively. By providing a robust solution to insider’s threat detection, the hybrid model offers organizations a promising approach to fortify security measures and safeguard against potential breaches.

An empirically based object-oriented testing using Machine learning

INTRODUCTION: The rapid growth of machine learning has the potential to revolutionize various industries and applications by automating complex tasks and enhancing efficiency. Effective software testing is crucial for ensuring software quality and minimizing resource expenses in software engineering. Machine learning techniques play a vital role in software testing by aiding in test case prioritization, predicting software defects, and analyzing test results. OBJECTIVES: The primary objective of this study is to explore the use of machine learning algorithms for software defect prediction. METHODS: Machine Learning models including Random Forest Classifier, Logistic Regression, K Nearest Neighbors, Gradient Boosting Classifiers, Catboost Classifier, and Convolutional Neural Networks have been employed for the study. The dataset includes a wide range of features relevant to software defect prediction and evaluates the performance of different prediction models. The study also focussed on developing hybrid models using stacking classifiers, which combine multiple individual models to improve accuracy. RESULTS: The experimental results show that the hybrid models combining CatBoost and Convolutional Neural Network have outperformed individual models, achieving the highest accuracy of 89.5%, highlighting the effectiveness of combining machine learning algorithms for software defect prediction. CONCLUSION: In conclusion, this study sheds light on the pivotal role of machine learning in enhancing software defect prediction.

A Hybrid System for Defect Detection on Rail Lines through the Fusion of Object and Context Information.

Defect detection on rail lines is essential for ensuring safe and efficient transportation. Current image analysis methods with deep neural networks (DNNs) for defect detection often focus on the defects themselves while ignoring the related context. In this work, we propose a fusion model that combines both a targeted defect search and a context analysis, which is seen as a multimodal fusion task. Our model performs rule-based decision-level fusion, merging the confidence scores of multiple individual models to classify rail-line defects. We call the model "hybrid" in the sense that it is composed of supervised learning components and rule-based fusion. We first propose an improvement to existing vision-based defect detection methods by incorporating a convolutional block attention module (CBAM) in the you only look once (YOLO) versions 5 (YOLOv5) and 8 (YOLOv8) architectures for the detection of defects and contextual image elements. This attention module is applied at different detection scales. The domain-knowledge rules are applied to fuse the detection results. Our method demonstrates improvements over baseline models in vision-based defect detection. The model is open for the integration of modalities other than an image, e.g., sound and accelerometer data.

Randomness-restricted Diffusion Model for Ocular Surface Structure Segmentation.

Ocular surface diseases affect a significant portion of the population worldwide. Accurate segmentation and quantification of different ocular surface structures are crucial for the understanding of these diseases and clinical decision-making. However, the automated segmentation of the ocular surface structure is relatively unexplored and faces several challenges. Ocular surface structure boundaries are often inconspicuous and obscured by glare from reflections. In addition, the segmentation of different ocular structures always requires training of multiple individual models. Thus, developing a one-model-fits-all segmentation approach is desirable. In this paper, we introduce a randomness-restricted diffusion model for multiple ocular surface structure segmentation. First, a time-controlled fusion-attention module (TFM) is proposed to dynamically adjust the information flow within the diffusion model, based on the temporal relationships between the network's input and time. TFM enables the network to effectively utilize image features to constrain the randomness of the generation process. We further propose a low-frequency consistency filter and a new loss to alleviate model uncertainty and error accumulation caused by the multi-step denoising process. Extensive experiments have shown that our approach can segment seven different ocular surface structures. Our method performs better than both dedicated ocular surface segmentation methods and general medical image segmentation methods. We further validated the proposed method over two clinical datasets, and the results demonstrated that it is beneficial to clinical applications, such as the meibomian gland dysfunction grading and aqueous deficient dry eye diagnosis.

Link Prediction through Ensemble Techniques

Link prediction is significant in the field of network analysis which helps to determine the future or hidden connections between the entities in a network. Traditional link prediction methods typically utilize single algorithms to make predictions, which might not be optimal for diverse and complex networks. In this research paper, a new framework is proposed which utilizes the existing link prediction algorithms with ensemble techniques. We have explored three ensemble strategies: bagging, boosting, and voting classifier and four individual link prediction algorithms: Adamic Adar, Katz, CCPA, Accurate link prediction. Methods considering ensemble techniques combine multiple individual models to improve prediction accuracy and robustness. We have experimentally compared the performance of proposed framework using six real world datasets in terms of AUC and Accuracy. The results show the superiority of the proposed framework over individual link prediction algorithms.

Tumor Segmentation in Colorectal Ultrasound Images Using an Ensemble Transfer Learning Model: Towards Intra-Operative Margin Assessment

Tumor boundary identification during colorectal cancer surgery can be challenging, and incomplete tumor removal occurs in approximately 10% of the patients operated for advanced rectal cancer. In this paper, a deep learning framework for automatic tumor segmentation in colorectal ultrasound images was developed, to provide real-time guidance on resection margins using intra-operative ultrasound. A colorectal ultrasound dataset was acquired consisting of 179 images from 74 patients, with ground truth tumor annotations based on histopathology results. To address data scarcity, transfer learning techniques were used to optimize models pre-trained on breast ultrasound data for colorectal ultrasound data. A new custom gradient-based loss function (GWDice) was developed, which emphasizes the clinically relevant top margin of the tumor while training the networks. Lastly, ensemble learning methods were applied to combine tumor segmentation predictions of multiple individual models and further improve the overall tumor segmentation performance. Transfer learning outperformed training from scratch, with an average Dice coefficient over all individual networks of 0.78 compared to 0.68. The new GWDice loss function clearly decreased the average tumor margin prediction error from 1.08 mm to 0.92 mm, without compromising the segmentation of the overall tumor contour. Ensemble learning further improved the Dice coefficient to 0.84 and the tumor margin prediction error to 0.67 mm. Using transfer and ensemble learning strategies, good tumor segmentation performance was achieved despite the relatively small dataset. The developed US segmentation model may contribute to more accurate colorectal tumor resections by providing real-time intra-operative feedback on tumor margins.

GERPM: A Geographically Weighted Stacking Ensemble Learning-Based Urban Residential Rents Prediction Model

Accurate prediction of urban residential rents is of great importance for landlords, tenants, and investors. However, existing rents prediction models face challenges in meeting practical demands due to their limited perspectives and inadequate prediction performance. The existing individual prediction models often lack satisfactory accuracy, while ensemble learning models that combine multiple individual models to improve prediction results often overlook the impact of spatial heterogeneity on residential rents. To address these issues, this paper proposes a novel prediction model called GERPM, which stands for Geographically Weighted Stacking Ensemble Learning-Based Urban Residential Rents Prediction Model. GERPM comprehensively analyzes the influencing factors of residential rents from multiple perspectives and leverages a geographically weighted stacking ensemble learning approach. The model combines multiple machine learning and deep learning models, optimizes parameters to achieve optimal predictions, and incorporates the geographically weighted regression (GWR) model to consider spatial heterogeneity. By combining the strengths of deep learning and machine learning models and taking into account geographical factors, GERPM aims to improve prediction accuracy and provide robust predictions for urban residential rents. The model is evaluated using housing data from Nanjing, a major city in China, and compared with representative individual prediction models, the equal weight combination model, and the ensemble learning model. The experimental results demonstrate that GERPM outperforms other models in terms of prediction performance. Furthermore, the model’s effectiveness and robustness are validated by applying it to other major cities in China, such as Shanghai and Hangzhou. Overall, GERPM shows promising potential in accurately predicting urban residential rents and contributing to the advancement of the rental market.

앙상블 비모수적 변수 선택 방법

Ensemble techniques are a method of generalizing multiple individual models to one optimal model, showing high predictive performance. It has the advantage of being available in both regression and classification problems, but it has the disadvantage that it is difficult to interpret the model due to its black-box characteristics. These shortcomings are supplemented by presenting a measure of variable importance, but this is only a relative value between explanatory variables, and there is a limit that the significance of each variable cannot be confirmed. On the other hand, in the case of the regression model, it is possible to test the significance of the variable through p-value and penalty. This study proposed a nonparametric variable selection method that tests the significance of explanatory variables based on the variable importance given in the ensemble technique. The simulation was conducted with eight models, and the results of sensitivity and specificity were compared by methodology. As a result of the simulation, the ensemble nonparametric variable selection showed better classification performance than the variable selection in the existing regression model in eight models. In addition, through case analysis, it was confirmed that the random forest nonparametric variable selection showed excellent performance even when there was a strong correlation between variables.

An adaptive ensemble framework with representative subset based weight correction for short-term forecast of peak power load

In the ensemble learning-based load prediction systems, recent research studies have achieved remarkable results in terms of extracting the complementary merits of multiple individual models and reducing the forecasting uncertainty, but they lack appropriate guidance for the sample-specific ensemble weighting of actual load data showing dynamics and variability. To this end, this study proposes an adaptive ensemble framework to address the above challenging and still tricky problem, and an adaptive weight-based ensemble learning model (AW-ELM) is framed to forecast short-term electric load by introducing the representative subset based weight correction with respect to the data characteristics. In the model, a weight library is formed as representatives extracting from the given training data. Based on the representatives, probability distribution modeling is built via a distance method. Then, the adaptive weights of the test data are determined by using the best matching concept. Experiments of two electric power companies in Jiangxi Province, China are implemented to evaluate the performance of the ensemble model. The results show that the ensemble model achieves better forecasting accuracy than the average combined forecasting model and the comparison models.

A forecasting system for deterministic and uncertain prediction of air pollution data

Air quality forecasting has practical implications for policy-makers guiding pollution control at early stages. However, due to the complexity and nonlinearity of air quality data, it is difficult to improve the forecasting precision of air quality indices by a single forecasting model. Accordingly, a novel combined model integrating decomposition and reconstructing techniques and multiple individual models is proposed to produce both point and interval forecasts for air quality data. In the devised model, first, the original series is decomposed into several independent intrinsic mode functions and a residue by an algorithm based on the complete ensemble empirical mode decomposition method. Then, the obtained modes are reconstructed into components by an improved version of the run-length judgment method and the number of reconstructed components is optimized to make a tradeoff between a higher forecasting accuracy and a lower workload in the designed system. To forecast different frequency components, various individual models are utilized. Finally, according to the results of point forecasts, the distribution of forecasting error is analyzed and interval forecasts are developed by the quantile regression method to reflect more uncertain information about the analyzed air pollution data. The concentration series of four major air pollutants in Hefei are selected as examples to test the robustness and effectiveness of the proposed combined model. The results indicate that, on the one hand, the established combined model outperforms three individual models and five hybrid models, and on the other hand, it can offer more valuable suggestions for policy-makers.

VIX term structure forecasting: New evidence based on the realized semi-variances

Considering the asymmetric volatility response to positive and negative shocks, this paper investigates VIX term structure forecasting by incorporating the realized upside and downside semi-variances based on high-frequency data, named good volatility and bad volatility, into the discrete-time GARCH-type model. We derive the risk-neutral model specification and calculate the analytical expression of VIX term structure, then estimate parameters by the maximum likelihood method. The MoP strategy (momentum of predictability, Wang et al., 2018) is extended for VIX term structure forecasting. Our empirical results show that incorporating high frequency data and considering the asymmetric shocks of realized variance are necessary to get much more accurate forecasting. The application of MoP strategy demonstrates the superior forecasting ability of integrating the advantages of multiple individual models. The evaluation of economic significance further confirms the superiority of our newly proposed model and the combination of multiple individual models, and results are robust under the alternative realized measure.

A novel class of reliability-based parallel hybridization (RPH) models for time series forecasting

Hybridization of individual models emerged as a predominant alternative for increasing accuracy in time series forecasting. The literature is abundant on providing hybrid methods aiming at improving forecasting accuracy and comprehensive pattern recognition. The principle behind all hybrid models' success is improving forecasting accuracy. One of the most widely used combination methods in line with the common objective of the hybridization concept is a parallel approach in which the forecasting model applied on the original time series and the weighted forecasts generate the final hybrid result. In parallel hybridization, the key is how to select the weights of each model. The weighting algorithm plays a significant role and the degree of accuracy in such models directly depends on it. However, the traditional parallel hybrid models developed to enhance forecasting accuracy employing multiple individual models simultaneously, faced some limitations, e.g., missing considering the reliability and generalization criteria of developed hybrid models. From the literature, it is observed that none of the parallel hybrid modes proposed in the literature focused on the improving reliability of the hybrid model to obtain better model generalization for unseen data. Thus, the main objective of this study is to propose a novel class of hybrid models named reliability-based parallel hybridization (RPH). The main goal of RPH methodology is to improve hybrid models' reliability rather than accuracy using a new reliable-based weighting algorithm (RWA). The RWA instead of traditional accuracy-based weighting algorithms in which error measurements are minimized, the reliability of hybrid models is maximized to determine the exact optimum weights of individual models. The RWA computed the optimum weights of forecasts regarding minimizing performance changing of the hybrid model in the validation data set. The RPH model proposed two crucial concepts for the first time: (1) Bringing up reliability in hybridization procedure (2) Proposing a novel reliable-based weighting algorithm to maximize the generalization power of the hybrid model. The bi-component version of the RPH model is proposed in this paper using ARIMA and MLP models. The forecasting power of the proposed PRH constructed based on ARIMA and MLP models is verified using the benchmark data sets e.g. the closing of Standard and Poor's 500 indexes (S&P500), the closing of the Shenzhen Integrated Index (SZII), and the opening of the Dow Jones Industrial Average Index (DJIAI). The experimental results indicate that the forecasting performance of the RPH model is much better than traditional accuracy-based parallel hybrid models.

Short-term electricity consumption prediction for buildings using data-driven swarm intelligence based ensemble model

Short-term building energy usage prediction plays an important role in fields of building energy management, power plants dispatch, and peak demand conflicting with grid security. A large number of data-driven models were applied to building and larger scale energy consumption prediction in the past two decades. Although successes have been achieved by these models for specific cases, no single model has dominated others over all cases. To improve model’s prediction performance in applications to different cases, a scheme of ensemble learning is proposed in this study. This ensemble scheme includes a self-adaptive model package that fuses the characteristics of multiple individual models together. Total five swarm intelligence based data-driven models are chosen as primary predictors of the ensemble scheme. The recursive feature elimination (RFE) is used for essential feature selection, and the K-fold cross validation method is applied to avoid over-fitting problem. Two sets of real buildings’ electricity usage data are collected for the performance comparison. Case A is from energy prediction shooting contest I organized by the American Society of Heating Refrigerating and Air-conditioning Engineer (ASHRAE) and Case B is from a campus building at the University of Wyoming, USA. The results show that the accuracy of the proposed ensemble model is better than that of any individual base model in both cases. Due to the generalization ability, the proposed ensemble model has the potential to be the unified model for different building energy consumption prediction cases.

Ensemble modelling in descriptive epidemiology: burden of disease estimation.

Ensemble modelling is a quantitative method that combines information from multiple individual models and has shown great promise in statistical machine learning. Ensemble models have a theoretical claim to being models that make the 'best' predictions possible. Applications of ensemble models to health research have included applying ensemble models like the super learner and random forests to epidemiological prediction tasks. Recently, ensemble methods have been applied successfully in burden of disease estimation. This article aims to provide epidemiologists with a practical understanding of the mechanisms of an ensemble model and insight into constructing ensemble models that are grounded in the epidemiological dynamics of the prediction problem of interest. We summarize the history of ensemble models, present a user-friendly framework for conceptualizing and constructing ensemble models, walk the reader through a tutorial of applying the framework to an application in burden of disease estimation, and discuss further applications.

Network representation learning with ensemble methods

This paper investigates network representation learning which involves network structures and labels. Most methods proposed so far try to utilize different kinds of network related data available in just one perfect model to learn the set of perfect embeddings, and then evaluate its performance comparing with other methods for downstream applications, such as node classification or link prediction. In this paper, we introduce ensemble learning to the study of network representation learning. In conventional scenario, ensemble methods train multiple individual learners and combine all their results about the same input to produce an improved one. Inspired by this, we try to train multiple individual models using network related data, and aggregate all the learned embeddings as the final network representations, to expect a performance boost for downstream applications. In order to learn good and diverse individual models, bootstrap sampling is used and different individual model structures are designed. Experimental results show that the idea of aggregating network embeddings really works, and can outperform existing excellent methods under specific experimental setups.

Predicting air quality with deep learning LSTM: Towards comprehensive models

In this paper we approach the problem of predicting air quality in the region of Madrid using long short term memory recurrent artificial neural networks. Air quality, in this study, is represented by the concentrations of a series of air pollutants which are proved as risky for human health such as CO, NO2, O3, PM10, SO2 and airborne pollen concentrations of two genus (Plantago and Poaceae). These concentrations are sampled in a set of locations in the city of Madrid. Instead of training an array of models, one per location and pollutant, several comprehensive deep network configurations are compared to identify those which are able to better extract relevant information out of the set of time series in order to predict one day-ahead air quality. The results, supported by statistical evidence, indicate that a single comprehensive model might be a better option than multiple individual models. Such comprehensive models represent a successful tool which can provide useful forecasts that can be thus applied, for example, in managerial environments by clinical institutions to optimize resources in expectation of an increment of the number of patients due to the exposure to low air quality levels.

11 LEVERAGING PHENOTYPE DATA TO IDENTIFY POTENTIAL CARRIERS OF DELETERIOUS GENETIC VARIANTS

This paper investigates network representation learning which involves network structures and labels. Most methods proposed so far try to utilize different kinds of network related data available in just one perfect model to learn the set of perfect embeddings, and then evaluate its performance comparing with other methods for downstream applications, such as node classification or link prediction. In this paper, we introduce ensemble learning to the study of network representation learning. In conventional scenario, ensemble methods train multiple individual learners and combine all their results about the same input to produce an improved one. Inspired by this, we try to train multiple individual models using network related data, and aggregate all the learned embeddings as the final network representations, to expect a performance boost for downstream applications. In order to learn good and diverse individual models, bootstrap sampling is used and different individual model structures are designed. Experimental results show that the idea of aggregating network embeddings really works, and can outperform existing excellent methods under specific experimental setups.