Generalization Capacity Research Articles

Feature selection for indoor temperature prediction in large-space buildings based on transfer entropy and life cycle cost

Indoor temperature prediction as an essential reference for heating, ventilation, and air-conditioning (HVAC) system tuning has boomed in recent years, with the development of information technology. But for data-driven techniques, inputs as key factors in prediction accuracy, learning efficiency and generalization capacity, did not pay enough attention. Existing research focused only on prediction accuracy by copying previous practices in other fields like medicine and hydrology, without considering prediction horizons and economic costs, which is meaningless and even may mislead the configuration of the monitoring system, especially for large-space buildings with long-span space, diverse zones and coupling parameters. In this work, a novel hybrid feature selection technique for indoor temperature prediction was developed. Herein, transfer entropy (TE) was introduced into filter to evaluate importance of features, where Max-min TE as a gauge was defined to cope with time delays in HVAC system; Then gated recurrent network was taken in wrapper as a bridge to transform evaluations in filter into contributions to prediction; Lastly, life cycle cost was employed as a referee to judge cost-efficient alternatives. The proposed technique was validated by time-series data collected from a library and system energy consumption baseline simulated with TRNSYS to suggest optimal feature subsets under different prediction horizons.

A Malicious Code Detection Method Based on Stacked Depthwise Separable Convolutions and Attention Mechanism.

To address the challenges of weak model generalization and limited model capacity adaptation in traditional malware detection methods, this article presents a novel malware detection approach based on stacked depthwise separable convolutions and self-attention, termed CoAtNet. This method combines the strengths of the self-attention module's robust model adaptation and the convolutional networks' powerful generalization abilities. The initial step involves transforming the malicious code into grayscale images. These images are subsequently processed using a detection model that employs stacked depthwise separable convolutions and an attention mechanism. This model effectively recognizes and classifies the images, automatically extracting essential features from malicious software images. The effectiveness of the method was validated through comparative experiments using both the Malimg dataset and the augmented Blended+ dataset. The approach's performance was evaluated against popular models, including XceptionNet, EfficientNetB0, ResNet50, VGG16, DenseNet169, and InceptionResNetV2. The experimental results highlight that the model surpasses other malware detection models in terms of accuracy and generalization ability. In conclusion, the proposed method addresses the limitations of traditional malware detection approaches by leveraging stacked depthwise separable convolutions and self-attention. Comprehensive experiments demonstrate its superior performance compared to existing models. This research contributes to advancing the field of malware detection and provides a promising solution for enhanced accuracy and robustness.

A Model Based on the Random Forest Algorithm That Predicts the Total Oil–Water Two-Phase Flow Rate in Horizontal Shale Oil Wells

Due to variables like wellbore deviation variation and flow rate, the local flow velocity in the output wellbore of horizontal shale oil wells varied significantly at various points in the wellbore cross-section, making it challenging to calculate the total single-layer production with accuracy. The oil–water two-phase flow rate calculation techniques for horizontal wells developed based on particular flow patterns and array spinners had excellent applicability in their respective niches but suffered from poor generalizability and demanding experience levels for logging interpreters. In this study, we employed five spinners in a triangular walled array instrument to create the multi-decision tree after figuring out how many leaf nodes there were and examining the defining characteristics of the observed values gathered under various experimental setups. The construction of the entire oil–water two-phase flow prediction model was made possible when the random forest regression approach was used with it. The total oil–water flow rate at each perforated layer was predicted using the model in sample wells, and the mean square error with the third party’s interpretation conclusion was 1.42, indicating that the model had an excellent application effect. The approach, which offered a new interpretation method for calculating the oil–water two-phase flow rate of horizontal wells based on multi-location local flow rate, required less interpretation knowledge from the interpreter and had a stronger generalization capacity.

GW-net: An efficient grad-CAM consistency neural network with weakening of random erasing features for semi-supervised person re-identification

Person re-identification (Re-ID) subject to loss of detailed information (sacrificing certain intricate body details) in samples caused by conventional regularization algorithms and degraded cross-domain performance due to limited generalization capacity is a challenging task. The majority of the existing research efforts address this problem either for attention regularization or cross-domain task, but neglect to explore a powerful framework to consider solving both cases simultaneously. To overcome this limitation, this paper develops an efficient semi-supervised person Re-ID network with Grad-CAM consistency regularization and weakening of random erasing features (GW-Net) to explore rich features and expect non-degraded performance in cross-domain condition. Specifically, a Grad-CAM consistency regularization (GCCR) module is designed to capture detailed information by using Grad-CAM for consistency regularization, thereby enhancing the capacity of intra-class feature mining. Secondly, a weakening of random erasing features (WREF) module is presented to reduce the impact of erased regions on texture features, thereby maintaining the performance on the general source domain while preventing performance degradation in the target domain. Thirdly, a Grad-CAM consistency regularization loss is introduced to enable our model to maintain consistency between Grad-CAM results of the unlabeled and its augmented images. Meanwhile, a feature consistency loss is reported to keep consistency between features of unlabeled samples. Finally, sufficient experiments are carried out on three representative datasets, which validate the efficacy and meliority of our presented approach over state-of-the-art methods.

Credit Default Prediction Based on Blending Learning Model

Currently, the preventions of credit default usually will be evaluated by users credit value before loaning from banks. However, for the loan user, who have no existing record of loaning and the situation of low credit value, it cannot precisely recognize the risk of credit default. After a credit default, the bank not only doesnt get the signed compensation and principal in time, but also the debtor needs to bear the expensive corresponding late fees and credit costs. Therefore, reducing credit defaults can decline more burden of debtors and creditors. In this paper, the authors evaluate multiple machine learning models including algorithms belong to machine learning and deep learning, using blending model to boost the prediction effect and accuracy, while proposing an optimization design to further enhance the stability, accuracy and generalization capacity of proposed algorithm, so as to effectively decrease the credit default rate and the risk of bank loss in practice.

Highly effective end-to-end single-to-multichannel feature fusion and ensemble classification to decode emotional secretes from small-scale spontaneous facial micro-expressions

Facial Micro-Expression (ME) is one of the pre-dominating non-verbal clues to demystify the true emotional states that people try to conceal cautiously. But emotion recognition from spontaneous ME images limits the high accuracy due to short duration and low intensity, lack of sufficient samples and consistencies among publicly available ME datasets. In this study, we have proposed two highly effective, lightweight, and generalized single-channel DLRRF-MER, and multi-channel DLH-3C-FUSION fusion models inspired by deep dense convolutional models and texture-based feature descriptors Local Binary Pattern (LBP) and Histogram of Oriented Gradients (HOG) to recognize ME from apex frame by constructing a composite dataset from five publicly available ME datasets CASME, CASMEII, CAS(ME)2, SAMM and MMEW. Pre-training has been done on a new composition of five facial macro expressions datasets CK+, MUGFE, OuluCasia, SFEW, and RAF-DB. The proposed models are fine-tuned on the target ME dataset rigorously with Stratified 5-Fold and 10-Fold, Leave-One-Subject-Out, and Leave-One-Dataset-Out(LODO) cross-validations(CV). In all evaluations, our proposed algorithms show remarkable improvement in effectiveness which surpasses the state-of-the-art accuracies and results in higher generalization capacity.

FedFTHA: A Fine-Tuning and Head Aggregation Method in Federated Learning

Personalized federated learning is a sub-field of federated learning. Contrary to conventional federated learning that expects to find a general global model, personalized federated learning generates a personalized model adapted to its local data distribution for each client. Some existing personalized federated learning methods only consider improving the client-side personalization ability, discarding the server-side generalization capacity. To address this issue, we propose a fine-tuning and head aggregation method in federated learning (FedFTHA). It allows each client to maintain a personalized model head and fine-tune it after each local update to generate a local model containing the personalized head. Specifically, during FedFTHA training, these personalized heads are aggregated to generate a generalized head for the global model. FedFTHA meets the needs of both client-side model personalization and server-side model generalization. In addition, a universal optimization framework is employed to prove its convergence under convex and non-convex conditions. We verify the personalization ability and generalization performance of FedFTHA under heterogeneous settings with benchmark datasets. The comparative analysis authenticates the significance of our proposal.

Multi-Oriented Enhancement Branch and Context-Aware Module for Few-Shot Oriented Object Detection in Remote Sensing Images

For oriented object detection, the existing CNN-based methods typically rely on a substantial and diverse dataset, which can be expensive to acquire and demonstrate limited capacity for generalization when faced with new categories that lack annotated samples. In this case, we propose MOCA-Net, a few-shot oriented object detection method with a multi-oriented enhancement branch and context-aware module, utilizing a limited number of annotated samples from novel categories for training. Especially, our method generates multi-oriented and multi-scale positive samples and then inputs them into an RPN and the detection head as a multi-oriented enhancement branch for enhancing the classification and regression capabilities of the detector. And by utilizing the context-aware module, the detector can effectively extract contextual information surrounding the object and incorporate it into RoI features in an adaptive manner, thereby improving its classification capability. As far as we know, our method is the first to attempt this in this field, and comparative experiments conducted on the public remote sensing dataset DOTA for oriented object detection showed that our method is effective.

Object classification through unseen scatters with a physical model trained neural network

Deep learning (DL) has been extensively employed to imaging or classify objects through scattering media. However, the performance of a trained network degrades dramatically when it is tested at scattering conditions statistically differing from those in the training. Therefore, how to retain DL’s generalization capacity is still a challenging task, with imaging or classification through untrained (unseen) scatters is highly desired. Here, we introduce physical model approach for network training, which enables generalization capability via learning from sufficiently large amount of model-based synthetic speckles. By mimicking the point spread function (PSF) of an incoherent scattering system, and diversifying the scattering realizations with random phasers, a generalized neural network is obtained. Object classification through unseen scatters is experimentally demonstrated. The random phasers prohibit DL from converging to a specific training medium, which is crucial to the creation of a generalized neural network. Unlike traditional data-driven DL that has nearly zero predicting accuracy, our model-trained network is generalized and being able to classify objects through untrained scattering media with accuracy as high as 60%. Neither the experimental scenes nor system geometry is required in creating the synthetic speckles, our method represents a genuine approach in data generation for DL training, paving the way to a highly generalized neural network for inverse scattering problems.

MTBERT-Attention: An Explainable BERT Model based on Multi-Task Learning for Cognitive Text Classification

In recent years, there has been a lot of focus on Bloom’s taxonomy-based classification of E-Learning materials. Researchers employ different methods and features. In our previous works, we have dealt with this problem via different techniques and algorithms. we started by boosting traditional machine learning algorithms then we moved on to deep learning algorithms by proposing a bidirectional GRU (BI-GRU) model combined with word2vec, finally we also proposed a fine-tuned model of bidirectional encoder representations from Transformers (BERT). The limitations of our model’s performance as well as the lack of an annotated dataset lead us to explore a novel approach to the cognitive classification of text. First, we propose MTBERT-Attention, a unique and explainable model based on multi-task learning (MTL), BERT, and the co-attention mechanism. MTL enhances our primary task’s generalization capacity and permits data augmentation. The usage of BERT as a stack for knowledge transfer between activities has been improved. The co-attention mechanism put particular emphasis on the significant aspects of the learning objective. We propose an explainability framework based on the attention mechanism. Lastly, we carry out in-depth tests to assess the viability and efficiency of the suggested model and the explainability Framework. Our proposed model outperforms the baseline models for loss, F1-score, and accuracy. It attained an overall classification accuracy of 97.71% with the test set and effectively classifies learning objectives that use unclear action verbs from Bloom’s taxonomy. On the other hand, to prove the performance of our explainability Framework, we conduct a qualitative and quantitative study on the quality of the explanations as well as on the computational cost. We adopt Local Interpretable Model-agnostic Explanations (LIME) as a baseline for comparison. Our experiments show that our proposed approach for explainability outperforms the LIME explainer in terms of fidelity and computational cost.

Predicting permanent strain accumulation of unbound aggregates using machine learning algorithms

The objective of this study is to propose a machine learning-based framework for predicting the permanent strain accumulation of unbound aggregates. To develop the machine learning-based framework, material properties of 15 crushed aggregates determined at two different gradations are combined with the applied stress states on aggregate specimens during repeated load triaxial testing by considering the aggregate shear strength characteristics. Three single machine learning algorithms (K-nearest neighbor, neural network, and decision tree) and two ensemble learning algorithms (random forest and extreme gradient boosting) are used in this study. Based on the dataset obtained from 90 permanent deformation tests, hyperparameters, which are used to control the learning process, are optimized. The numerical results demonstrate that all five of the tuned machine learning models exhibit good generalization capacity and performance with the coefficient of determination exceeding 0.99. Among the five models, the extreme gradient boosting model outperforms the others, accurately predicting permanent strain accumulation even for load cycles of 10,000. The contributions of individual input features to the performance of each learning model depend on the dataset. The strength characteristics and the applied deviatoric stress are the two most important features. Therefore, the machine learning-based framework introduced in this study may be effectively used for predicting the permanent strain accumulation of unbound aggregate layer.

Deep learning-based prediction of steady surface settlement due to shield tunnelling

Predicting ground movement produced by shield tunnelling in densely built urban areas is of practical significance. This study introduces an artificial intelligence method to predict the steady surface settlement caused by shield tunnelling. The proposed method comprises three models: a hybrid deep neural network model (HDNN), an error correction model (ECM), and an auxiliary model. Based on the settlement mechanism associated with tunnelling, the factors related to the ground movement were classified into five categories. The first four types of factors were used as the input parameters of the HDNN to predict steady surface settlement. The ECM considers unmeasurable factors. Five datasets from different subway lines were used to assess the performance and generalisation capacity of the proposed method. The results show that the proposed method outperforms classical deep neural networks as it considers the settlement mechanism and spatial and temporal characteristics of the input parameters.

Application of black-box models based on artificial intelligence for the prediction of chlorine and TTHMs in the trunk network of Bogotá, Colombia

Abstract The chlorine and total trihalomethane (TTHM) concentrations are sparsely measured in the trunk network of Bogotá, Colombia, which leads to a high uncertainty level at an operational level. For this reason, this research assessed the prediction accuracy for chlorine and TTHM concentrations of two black-box models based on the following artificial intelligence techniques: artificial neural networks (ANNs) and adaptive neuro-fuzzy inference system (ANFIS) as a modelling alternative. The simulation results of a hydraulic and water quality analysis of the network in EPANET and its multi-species extension EPANET-MSX were used for training the black-box models. Subsequently, the Threat Ensemble Vulnerability Assessment-Sensor Placement Optimization Tool (TEVA-SPOT) and Evolutionary Polynomial Regression-Multi-Objective Genetic Algorithm (EPR-MOGA-XL) were jointly applied to select the most representative input variables and locations for predicting water quality at other points of the network. ANNs and ANFIS were optimized with a multi-objective approach to reach a compromise between training performance and generalization capacity. The ANFIS models had a higher mean Training and Test Nash–Sutcliffe Index (NSI) in contrast with ANNs. In general, the models had a satisfactory mean prediction performance. However, some of them did not achieve suitable Test NSI values, and the prediction accuracy for different operational statuses was limited.

A machine learning-based genetic programming approach for the sustainable production of plastic sand paver blocks

Plastic sand paver blocks (PSPB) provide a sustainable alternative by reprocessing plastic waste and decreasing reliance on environmentally hazardous materials such as concrete. They promote waste management and environmentally favorable building practices. This paper presents a novel method for estimating the compressive strength (CS) of plastic sand paver blocks based on gene expression programming (GEP) techniques. The database collected from the experimental work comprises 135 compressive strength results. Seven input parameters were involved in predicting the CS of PSPB, namely, plastic, sand, sand size, fiber percentage, fibre length, fibre diameter, and tensile strength of the fibre. Simplified mathematical expressions were used to figure out the CS. The results of GEP formulations showed that they were better in line with the experimental data, with R2 values for CS of 0.89 (training) and 0.88 (testing). The models' performance was evaluated using sensitivity analysis and statistical checks. The statistical evaluations show that the actual and predicted values are closer together, which lends credence to the GEP model's capacity to forecast PSPB CS. The sensitivity analysis showed that sand size and fibre percentage contribute more than 50% of the CS in PSPB. In addition, the results demonstrate that the proposed models are accurate and have a robust capacity for generalization and prediction. This research can improve environmental protection and economic benefit by enhancing the reuse of PSPB in producing green ecosystems.

Hybrid machine learning techniques in the management of harmful algal blooms impact

Harmful algal blooms (HABs) are episodes of high concentrations of algae that are potentially toxic for human consumption. Mollusc farming can be affected by HABs because, as filter feeders, they can accumulate high concentrations of marine biotoxins in their tissues. To avoid the risk to human consumption, harvesting is prohibited when toxicity is detected. At present, the closure of production areas is based on expert knowledge and the existence of a predictive model would help when conditions are complex and sampling is not possible. Although the concentration of toxin in meat is the method most commonly used by experts in the control of shellfish production areas, it is rarely used as a target by automatic prediction models. This is largely due to the irregularity of the data due to the established sampling programs. As an alternative, the activity status of production areas has been proposed as a target variable based on whether mollusc meat has a toxicity level below or above the legal limit. This new option is the most similar to the actual functioning of the control of shellfish production areas. For this purpose, we have made a comparison between hybrid machine learning models like Neural-Network-Adding Bootstrap (BAGNET) and Discriminative Nearest Neighbor Classification (SVM-KNN) when estimating the state of production areas. The study has been carried out in several estuaries with different levels of complexity in the episodes of algal blooms to demonstrate the generalization capacity of the models in bloom detection. As a result, we could observe that, with an average recall value of 93.41% and without dropping below 90% in any of the estuaries, BAGNET outperforms the other models both in terms of results and robustness.

Mind the Gap: Polishing Pseudo Labels for Accurate Semi-supervised Object Detection

Exploiting pseudo labels (e.g., categories and bounding boxes) of unannotated objects produced by a teacher detector have underpinned much of recent progress in semi-supervised object detection (SSOD). However, due to the limited generalization capacity of the teacher detector caused by the scarce annotations, the produced pseudo labels often deviate from ground truth, especially those with relatively low classification confidences, thus limiting the generalization performance of SSOD. To mitigate this problem, we propose a dual pseudo-label polishing framework for SSOD. Instead of directly exploiting the pseudo labels produced by the teacher detector, we take the first attempt at reducing their deviation from ground truth using dual polishing learning, where two differently structured polishing networks are elaborately developed and trained using synthesized paired pseudo labels and the corresponding ground truth for categories and bounding boxes on the given annotated objects, respectively. By doing this, both polishing networks can infer more accurate pseudo labels for unannotated objects through sufficiently exploiting their context knowledge based on the initially produced pseudo labels, and thus improve the generalization performance of SSOD. Moreover, such a scheme can be seamlessly plugged into the existing SSOD framework for joint end-to-end learning. In addition, we propose to disentangle the polished pseudo categories and bounding boxes of unannotated objects for separate category classification and bounding box regression in SSOD, which enables introducing more unannotated objects during model training and thus further improves the performance. Experiments on both PASCAL VOC and MS-COCO benchmarks demonstrate the superiority of the proposed method over existing state-of-the-art baselines. The code can be found at https://github.com/snowdusky/DualPolishLearning.

MSDC: Exploiting Multi-State Power Consumption in Non-intrusive Load Monitoring Based on a Dual-CNN Model

Non-intrusive load monitoring (NILM) aims to decompose aggregated electrical usage signal into appliance-specific power consumption and it amounts to a classical example of blind source separation tasks. Leveraging recent progress on deep learning techniques, we design a new neural NILM model {\em Multi-State Dual CNN} (MSDC). Different from previous models, MSDC explicitly extracts information about the appliance's multiple states and state transitions, which in turn regulates the prediction of signals for appliances. More specifically, we employ a dual-CNN architecture: one CNN for outputting state distributions and the other for predicting the power of each state. A new technique is invented that utilizes conditional random fields (CRF) to capture state transitions. Experiments on two real-world datasets REDD and UK-DALE demonstrate that our model significantly outperform state-of-the-art models while having good generalization capacity, achieving 6%-10% MAE gain and 33%-51% SAE gain to unseen appliances.

Boosting the performance of SOTA convolution-based networks with dimensionality reduction: An application on hyperspectral images of wine grape berries

Precision viticulture is an area that is very dependent on methods that allow for a sustainable assessment of grape maturity and, in this work, we apply two state-of-the-art (SOTA) convolution-based networks, namely InceptionTime and OmniScale 1D-CNN, to hyperspectral images of wine grape berries to estimate sugar content. Since attaining generalization capacity and processing the information in such high-dimensional data are the two biggest challenges to overcome in problems of this nature, we also study the impact of two dimensionality reduction techniques, Principal Component Analysis (PCA) and t-Distributed Stochastic Neighbor Embedding (t-SNE), on the models' performance. Both models underwent different tests with different vintages and varieties of wine grapes in the training/validation steps, as to form a true test to their generalization capacity. Our results show that both PCA and t-SNE succeed in improving the performance of these deep networks when an adequate number of components is chosen that minimizes the ratio between information loss and removing redundant features: additionally, both techniques significantly reduce computational cost, a very important trait when training deep learning models. Both models showed good generalization ability with very competitive results across different varieties and vintages even despite their significant differences in variability, which is an indicator that a relationship between spectras can be found that is reflected on sugar content values.

A general data quality evaluation framework for dynamic response monitoring of long-span bridges

The structural health monitoring system (SHM) of long-span bridges inevitably produces low-quality data. It is important to evaluate the data quality and screen out normal data. Most existing studies on data anomaly detection have focused on abnormal data with abnormal waveforms in time domain. Usually there are pseudo-normal data in the SHM system, which seems normal in time domain. However, the pseudo-normal data of bridge dynamic responses may cause wrong dynamic characteristics identification. In this study, the existing data anomaly detection was further expanded to data quality evaluation, and both obvious abnormal and pseudo-normal data are evaluated. A general data quality evaluation framework for dynamic response monitoring of long-span bridge structures was proposed. The time–frequency characteristics of monitoring data were obtained by continuous wavelet transform, and the data quality was classified and evaluated by constructing and training a convolutional neural network (CNN). The proposed framework was verified by using acceleration monitoring data of a cable-stayed bridge. The results reveal that this framework solves the problem that CNN cannot detect pseudo-normal data through time-domain characteristics, and improves the accuracy of data quality evaluation by using time–frequency information of the monitoring data. The acceleration monitoring data of a suspension bridge and a single pylon cable-stayed bridge were used to demonstrate the feasibility of the cross-object application of the proposed framework. The evaluation accuracy for the acceleration data quality of the suspension bridge exceeded 96%. After the reinforced training of a small number of pseudo-normal data samples, the evaluation accuracy for the acceleration data quality of the cables of the cable-stayed bridge reached 96.8%. The framework shows a good generalization capacity and robustness in cross-object applications.

Exploiting image translations via ensemble self-supervised learning for Unsupervised Domain Adaptation

Unsupervised Domain Adaptation (UDA) aims to improve the generalization capacity of models when they are tested on a real-world target domain by learning a model on a source labeled domain. Recently, a UDA method was proposed that addresses the adaptation problem by combining ensemble learning with self-supervised learning. However, this method uses only the source domain to pretrain the model and employs a limited amount of classifiers to create target pseudo labels. To mitigate these deficiencies, in this work, we explore the usage of image translations in combination with ensemble learning and self-supervised learning. To increase the model’s exposure to more variable pretraining data, our method creates multiple diverse image translations, which encourages the learning of domain-invariant features, desired to increase generalization. With these image translations, we are able to learn translation-specific classifiers, which also allows to maximize the amount of ensemble’s classifiers resulting in more robust target pseudo labels. In addition, we propose to use the target domain in pretraining stage to mitigate source domain bias in the network. We evaluate our method on the standard UDA benchmarks, i.e., adapting GTA V and Synthia to Cityscapes, and achieve state-of-the-art results on the mIoU metric. Extensive ablation experiments are reported to highlight the advantageous properties of our UDA strategy.