Multi-stage Learning Research Articles

Multi-stage semi-supervised learning enhances white matter hyperintensity segmentation.

White matter hyperintensities (WMHs) are frequently observed on magnetic resonance (MR) images in older adults, commonly appearing as areas of high signal intensity on fluid-attenuated inversion recovery (FLAIR) MR scans. Elevated WMH volumes are associated with a greater risk of dementia and stroke, even after accounting for vascular risk factors. Manual segmentation, while considered the ground truth, is both labor-intensive and time-consuming, limiting the generation of annotated WMH datasets. Un-annotated data are relatively available; however, the requirement of annotated data poses a challenge for developing supervised machine learning models. To address this challenge, we implemented a multi-stage semi-supervised learning (M3SL) approach that first uses un-annotated data segmented by traditional processing methods ("bronze" and "silver" quality data) and then uses a smaller number of "gold"-standard annotations for model refinement. The M3SL approach enabled fine-tuning of the model weights with the gold-standard annotations. This approach was integrated into the training of a U-Net model for WMH segmentation. We used data from three scanner vendors (over more than five scanners) and from both cognitively normal (CN) adult and patients cohorts [with mild cognitive impairment and Alzheimer's disease (AD)]. An analysis of WMH segmentation performance across both scanner and clinical stage (CN, MCI, AD) factors was conducted. We compared our results to both conventional and transfer-learning deep learning methods and observed better generalization with M3SL across different datasets. We evaluated several metrics (F-measure, IoU, and Hausdorff distance) and found significant improvements with our method compared to conventional (p < 0.001) and transfer-learning (p < 0.001). These findings suggest that automated, non-machine learning, tools have a role in a multi-stage learning framework and can reduce the impact of limited annotated data and, thus, enhance model performance.

Robust multi-stage progressive autoencoder for hyperspectral anomaly detection

Recently, Autoencoders (AEs) have demonstrated remarkable performance in the field of hyperspectral anomaly detection, owing to their powerful capability in handling high-dimensional data. However, they often overlook the inherent global distribution characteristics and long-range dependencies in hyperspectral images (HSI). This oversight makes it challenging to accurately characterize and describe boundaries between different backgrounds and anomalies in complex HSI, thereby affecting detection accuracy. To address this issue, a robust multi-stage progressive autoencoder for hyperspectral anomaly detection (RMSAD) is proposed. Initially, a progressive multi-stage learning framework based on convolutional autoencoders is employed. This framework incrementally reveals and integrates deep contextual features along with their long-range dependencies in HSI, aiming to accurately characterize the background and anomalies. Subsequently, an innovative multi-scale fusion strategy is introduced at the intersections of each stage, reinforcing the learning and representation of background and global spatial details across multiple stages. Finally, by collectively extracting abnormal spatial information across stages, effectively reducing the tendency of autoencoders to reconstruct anomalies. This ensures the efficient restoration and replication of global textural details in HSI. The experimental results on the six HSI datasets demonstrate that the proposed RMSAD is superior to other state-of-the-art methods.

NASPrecision: Neural Architecture Search-Driven Multi-Stage Learning for surface roughness prediction in ultra-precision machining

Accurate surface roughness prediction is critical for ensuring high product quality, especially in sectors such as manufacturing, aerospace, and medical devices, where the smallest imperfections can compromise performance or safety. However, this is very challenging due to complex, non-linear interactions among variables, which is further exacerbated when working with limited and imbalanced datasets. Existing methods leveraging traditional machine learning algorithms require extensive domain knowledge for feature engineering and substantial human intervention for model selection. To address these issues, we propose a Neural Architecture Search (NAS)-Driven Multi-Stage Learning framework, named NASPrecision. This innovative approach autonomously identifies the most suitable features and models for various surface roughness prediction tasks and significantly enhances the performance by multi-stage learning. Our framework operates in three stages: (1) architecture search stage, employing NAS to automatically identify the most effective model architecture; (2) initial training stage, where we train the neural network for initial predictions; (3) refinement stage, where a subsequent model is appended to refine and capture subtle variations overlooked by the initial training stage. In light of limited and imbalanced datasets, we adopt a generative data augmentation technique to balance and generate new data by learning the underlying data distribution. We perform extensive experiments on three distinct real-world datasets, each associated with a different machining technique, and compare with various machine learning algorithms. The experimental results underscore the superiority of our framework, which achieves an average improvement of 25.7%, 22.7%, and 39.1% in terms of Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Standard Deviation of Residuals (STDR), respectively. This significant performance enhancement not only confirms the robustness of our framework but also establishes it as a generic solution for accurate surface roughness prediction. The success of this approach can lead to improved production efficiency and product quality in critical industries while also reducing the need for extensive domain knowledge and human intervention.

Proxy endpoints — bridging clinical trials and real world data

Objective:Disease severity scores, or endpoints, are routinely measured during Randomized Controlled Trials (RCTs) to closely monitor the effect of treatment. In real-world clinical practice, although a larger set of patients is observed, the specific RCT endpoints are often not captured, which makes it hard to utilize real-world data (RWD) to evaluate drug efficacy in larger populations. Methods:To overcome this challenge, we developed an ensemble technique which learns proxy models of disease endpoints in RWD. Using a multi-stage learning framework applied to RCT data, we first identify features considered significant drivers of disease available within RWD. To create endpoint proxy models, we use Explainable Boosting Machines (EBMs) which allow for both end-user interpretability and modeling of non-linear relationships. Results:We demonstrate our approach on two diseases, rheumatoid arthritis (RA) and atopic dermatitis (AD). As we show, our combined feature selection and prediction method achieves good results for both disease areas, improving upon prior methods proposed for predictive disease severity scoring. Conclusion:Having disease severity over time for a patient is important to further disease understanding and management. Our results open the door to more use cases in the space of RA and AD such as treatment effect estimates or prognostic scoring on RWD. Our framework may be extended beyond RA and AD to other diseases where the severity score is not well measured in electronic health records.

S2Former-OR: Single-Stage Bi-Modal Transformer for Scene Graph Generation in OR.

Scene graph generation (SGG) of surgical procedures is crucial in enhancing holistically cognitive intelligence in the operating room (OR). However, previous works have primarily relied on multi-stage learning, where the generated semantic scene graphs depend on intermediate processes with pose estimation and object detection. This pipeline may potentially compromise the flexibility of learning multimodal representations, consequently constraining the overall effectiveness. In this study, we introduce a novel single-stage bi-modal transformer framework for SGG in the OR, termed, S2Former-OR, aimed to complementally leverage multi-view 2D scenes and 3D point clouds for SGG in an end-to-end manner. Concretely, our model embraces a View-Sync Transfusion scheme to encourage multi-view visual information interaction. Concurrently, a Geometry-Visual Cohesion operation is designed to integrate the synergic 2D semantic features into 3D point cloud features. Moreover, based on the augmented feature, we propose a novel relation-sensitive transformer decoder that embeds dynamic entity-pair queries and relational trait priors, which enables the direct prediction of entity-pair relations for graph generation without intermediate steps. Extensive experiments have validated the superior SGG performance and lower computational cost of S2Former-OR on 4D-OR benchmark, compared with current OR-SGG methods, e.g., 3 percentage points Precision increase and 24.2M reduction in model parameters. We further compared our method with generic single-stage SGG methods with broader metrics for a comprehensive evaluation, with consistently better performance achieved. Our source code can be made available at: https://github.com/PJLallen/S2Former-OR.

Mathematical Model of Logistic Regression for Binary Classification. Part 2. Data Preparation, Learning and Testing Processes

This article reviews the theoretical aspects of logistic regression for binary data classification, including data preparation processes, training, testing, and model evaluation metrics. Requirements for input data sets are formulated, methods of coding categorical data are described, methods of scaling input features are defined and substantiated. A scheme for learning logistic regression using the gradient descent method has been developed to minimize the loss function by the appropriate adjustment of the weights of the features of the sample of objects intended for classification. Features of the construction of recurrent methods of classical and stochastic gradient descent are determined. The requirements for the organization of the data sample for the multi-stage learning model in order to avoid overtraining or undertraining of logistic regression are described. The scheme of testing the trained logistic regression is given and the main quality metrics of binary classification are described. The influence of the height of the classification threshold on the efficiency of logistic regression was noted. According to the results of the work, the directions of perspective research of logistic regression are outlined.

A Novel Accelerated Multistage Learning Control Mechanism via Virtual Performance Reduction.

This study uses a multistage learning mechanism concept to investigate the accelerated learning control for stochastic systems. In this mechanism, the learning iterations are divided into successive stages, with each stage comprising several iterations. The learning gain is constant in each stage to accelerate the learning process and decreases it from one stage to another to eliminate the noise effect asymptotically. The critical issue is determining the switching iteration when a new stage starts. This study resolves this issue by calculating a virtual performance index of the mean-squared input error and its estimated upper bound. Specifically, the ideal, practical, and improved multistage learning control schemes are proposed to determine the switching iteration and generate the learning gain sequence. The ideal scheme achieves the best performance at the cost of a large computation burden, and the practical scheme saves computation cost, but the performance is not excellent. The improved scheme significantly approximates the best performance by introducing additional stretching parameters to the performance index. Illustrative simulations are provided to verify the theoretical results.

Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculum

Augmenting large language models (LLMs) with external tools has emerged as a promising approach to extending the capability of LLMs. Although there are some works that employ open-source LLMs for the tool-learning task, most of them are trained in a controlled environment in which LLMs only learn to execute the human-provided tools. However, selecting proper tools from the large toolset is also a crucial ability for the tool-learning model to be applied in real-world applications. Existing methods usually directly employ self-instruction methods to train the model, which ignores differences in tool complexity. In this paper, we propose the Confucius a novel tool-learning framework to train LLM to use complicated tools in real-world scenarios, which contains two main phases: (1) We first propose a multi-stage learning method to teach the LLM to use various tools from an easy-to-difficult curriculum; (2) thenceforth, we propose the Iterative Self-instruct from Introspective Feedback (ISIF) to dynamically construct the dataset to improve the ability to use the complicated tool. Extensive experiments conducted on both controlled and real-world settings demonstrate the superiority of our tool-learning framework in the real-world application scenario compared to both tuning-free (e.g., ChatGPT, Claude) and tuning-based baselines (e.g., GPT4Tools).

A clustering multi-stage training transfer learning framework for cross simulation domain and experimental domain fault diagnosis

Deep learning methods have demonstrated remarkable achievements in the field of fault diagnosis for rotating machinery. However, their effectiveness heavily relies on high-quality labeled samples, which presents a significant challenge owing to the limited availability of such data in engineering applications. To address this realistic issue, we propose a novel simulation-driven transfer learning model called the clustering multi-stage training transfer learning framework (CMSTL) for fault diagnosis of rolling bearings. The fundamental concept of the proposed method is to utilize simulation data as a substitute for labeled actual device data and integrate the suggested clustering learning and multistage training strategies to extract domain-independent and fault-discriminative features from simulation and experimental domains. Specifically, the clustering learning strategy is embedded into the CMSTL model to encourage the feature extractor to acquire distinguishable features associated with different categories while eliminating domain-specific knowledge, which enables samples near the classification decision boundary to cluster towards their respective clustering centers. Additionally, the proposed multistage learning strategy leverages the model trained with a certain level of accuracy in the first stage to annotate actual device samples, thereby enhancing both the precision of the pseudo-label for real data and the overall training stability of the model. The effectiveness and superiority of the proposed method were validated using both artificially damaged and run-to-failure datasets. The comparative analysis results demonstrate that the CMSTL method exhibits a minimum 2.2% improvement in fault diagnosis accuracy and enhances the clustering capability compared to other advanced transfer learning fault diagnosis methods.

The Fittest Wins: A Multistage Framework Achieving New SOTA in ViZDoom Competition

This paper offers an integrated solution for First-Person Shooter (FPS) games to train agents with adaptive strategies. Solving such complex decision tasks requires generalization ability and adaptive strategies. We develop a framework using a novel Adaptive Strategic Control (ASC) algorithm combined with advanced techniques like the hindsight experience replay (HER), multi-agent reinforcement learning (MARL), and league training. The approach adopts a multi-stage learning scheme, consisting of learning a goal-conditioned navigation policy, then transferring to learn sophisticated shooting skills by playing against a league of players, and finally learning adaptive strategies. Our agent achieves the SOTA result in past ViZDoom AI Competitions, surpassing previous top-ranked agents (never seen during training) by a large margin. We provide comprehensive analysis and experiments to elaborate the effect of each component in affecting the agent performance and demonstrate that the proposed and adopted techniques are essential to achieve superior performance in ViZDoom Competition and potentially valuable for general end-to-end FPS games.

An early-curricular team learning activity to foster integration of biochemical concepts and clinical sciences in undergraduate medical education.

The ability to connect key concepts of biochemistry with clinical presentations is essential for the development of clinical reasoning skills and adaptive expertise in medical trainees. To support the integration of foundational and clinical sciences in our undergraduate health science curricula, we developed a small group active learning exercise during which interprofessional groups of students use clinical cases to explore the biochemistry, diagnostic strategy, and evidence-based treatment options of inborn errors of metabolism (IEM). We designed multistage learning modules consisting of (1.) low-fidelity case simulations of pediatric patients presenting with IEMs, (2.) guided group discussions on clinical biochemistry, differential diagnoses, and diagnostic strategies, (3.) oral presentations of clinical reasoning strategies, and (4.) discussion of relevant evidence-based medicine topics related to the cases. These modules Scientific Knowledge Integrated in Patient Presentations (SKIPPs) were added to a first-semester foundational sciences course serving five health professions programs. The assessment of learning outcomes by students and faculty shows that SKIPPs sessions are well-received activities that significantly improve trainees' ability to integrate foundational science concepts into clinical scenarios, to practice interprofessional teamwork and to develop clinical reasoning skills.

Multi-Stage Learning Framework Using Convolutional Neural Network and Decision Tree-Based Classification for Detection of DDoS Pandemic Attacks in SDN-Based SCADA Systems

Supervisory Control and Data Acquisition (SCADA) systems, which play a critical role in monitoring, managing, and controlling industrial processes, face flexibility, scalability, and management difficulties arising from traditional network structures. Software-defined networking (SDN) offers a new opportunity to overcome the challenges traditional SCADA networks face, based on the concept of separating the control and data plane. Although integrating the SDN architecture into SCADA systems offers many advantages, it cannot address security concerns against cyber-attacks such as a distributed denial of service (DDoS). The fact that SDN has centralized management and programmability features causes attackers to carry out attacks that specifically target the SDN controller and data plane. If DDoS attacks against the SDN-based SCADA network are not detected and precautions are not taken, they can cause chaos and have terrible consequences. By detecting a possible DDoS attack at an early stage, security measures that can reduce the impact of the attack can be taken immediately, and the likelihood of being a direct victim of the attack decreases. This study proposes a multi-stage learning model using a 1-dimensional convolutional neural network (1D-CNN) and decision tree-based classification to detect DDoS attacks in SDN-based SCADA systems effectively. A new dataset containing various attack scenarios on a specific experimental network topology was created to be used in the training and testing phases of this model. According to the experimental results of this study, the proposed model achieved a 97.8% accuracy rate in DDoS-attack detection. The proposed multi-stage learning model shows that high-performance results can be achieved in detecting DDoS attacks against SDN-based SCADA systems.

Progressive Learning of 3D Reconstruction Network From 2D GAN Data.

This paper presents a method to reconstruct high-quality textured 3D models from single images. Current methods rely on datasets with expensive annotations; multi-view images and their camera parameters. Our method relies on GAN generated multi-view image datasets which have a negligible annotation cost. However, they are not strictly multi-view consistent and sometimes GANs output distorted images. This results in degraded reconstruction qualities. In this work, to overcome these limitations of generated datasets, we have two main contributions which lead us to achieve state-of-the-art results on challenging objects: 1) A robust multi-stage learning scheme that gradually relies more on the models own predictions when calculating losses and 2) A novel adversarial learning pipeline with online pseudo-ground truth generations to achieve fine details. Our work provides a bridge from 2D supervisions of GAN models to 3D reconstruction models and removes the expensive annotation efforts. We show significant improvements over previous methods whether they were trained on GAN generated multi-view images or on real images with expensive annotations.

When Sparse Neural Network Meets Label Noise Learning: A Multistage Learning Framework.

Recent methods in network pruning have indicated that a dense neural network involves a sparse subnetwork (called a winning ticket), which can achieve similar test accuracy to its dense counterpart with much fewer network parameters. Generally, these methods search for the winning tickets on well-labeled data. Unfortunately, in many real-world applications, the training data are unavoidably contaminated with noisy labels, thereby leading to performance deterioration of these methods. To address the above-mentioned problem, we propose a novel two-stream sample selection network (TS3-Net), which consists of a sparse subnetwork and a dense subnetwork, to effectively identify the winning ticket with noisy labels. The training of TS3-Net contains an iterative procedure that switches between training both subnetworks and pruning the smallest magnitude weights of the sparse subnetwork. In particular, we develop a multistage learning framework including a warm-up stage, a semisupervised alternate learning stage, and a label refinement stage, to progressively train the two subnetworks. In this way, the classification capability of the sparse subnetwork can be gradually improved at a high sparsity level. Extensive experimental results on both synthetic and real-world noisy datasets (including MNIST, CIFAR-10, CIFAR-100, ANIMAL-10N, Clothing1M, and WebVision) demonstrate that our proposed method achieves state-of-the-art performance with very small memory consumption for label noise learning. Code is available at https://github.com/Runqing-forMost/TS3-Net/tree/master.

Collaborative Target Search With a Visual Drone Swarm: An Adaptive Curriculum Embedded Multistage Reinforcement Learning Approach.

Equipping drones with target search capabilities is highly desirable for applications in disaster rescue and smart warehouse delivery systems. Multiple intelligent drones that can collaborate with each other and maneuver among obstacles show more effectiveness in accomplishing tasks in a shorter amount of time. However, carrying out collaborative target search (CTS) without prior target information is extremely challenging, especially with a visual drone swarm. In this work, we propose a novel data-efficient deep reinforcement learning (DRL) approach called adaptive curriculum embedded multistage learning (ACEMSL) to address these challenges, mainly 3-D sparse reward space exploration with limited visual perception and collaborative behavior requirements. Specifically, we decompose the CTS task into several subtasks including individual obstacle avoidance, target search, and inter-agent collaboration, and progressively train the agents with multistage learning. Meanwhile, an adaptive embedded curriculum (AEC) is designed, where the task difficulty level (TDL) can be adaptively adjusted based on the success rate (SR) achieved in training. ACEMSL allows data-efficient training and individual-team reward allocation for the visual drone swarm. Furthermore, we deploy the trained model over a real visual drone swarm and perform CTS operations without fine-tuning. Extensive simulations and real-world flight tests validate the effectiveness and generalizability of ACEMSL. The project is available at https://github.com/NTU-UAVG/CTS-visual-drone-swarm.git.

Multi-Stage Image-Language Cross-Generative Fusion Network for Video-Based Referring Expression Comprehension.

Video-based referring expression comprehension is a challenging task that requires locating the referred object in each video frame of a given video. While many existing approaches treat this task as an object-tracking problem, their performance is heavily reliant on the quality of the tracking templates. Furthermore, when there is not enough annotation data to assist in template selection, the tracking may fail. Other approaches are based on object detection, but they often use only one adjacent frame of the key frame for feature learning, which limits their ability to establish the relationship between different frames. In addition, improving the fusion of features from multiple frames and referring expressions to effectively locate the referents remains an open problem. To address these issues, we propose a novel approach called the Multi-Stage Image-Language Cross-Generative Fusion Network (MILCGF-Net), which is based on one-stage object detection. Our approach includes a Frame Dense Feature Aggregation module for dense feature learning of adjacent time sequences. Additionally, we propose an Image-Language Cross-Generative Fusion module as the main body of multi-stage learning to generate cross-modal features by calculating the similarity between video and expression, and then refining and fusing the generated features. To further enhance the cross-modal feature generation capability of our model, we introduce a consistency loss that constrains the image-language similarity and language-image similarity matrices during feature generation. We evaluate our proposed approach on three public datasets and demonstrate its effectiveness through comprehensive experimental results.

Multi-Stage Classification Scheme to Optimize Medical Treatments

In most existing machine learning and deep learning settings, classification and regression prediction problems may be described as a process where the model output is based on a single-stage input. In most real-life scenarios achieving the desired medical state for the patient may involve dynamically solving drug prescription problems based on the input data at different stages, where each stage is a logical grouping such as timestep division, ICU stay, etc. Data at a given stage represents a recovery progression and can be fundamentally different from the datasets from the previous and future stages. Although A single model may solve the task, a multi-stage learning procedure may be more suitable. To solve this task, we propose an FNN-driven ensemble-based approach for predicting the medications that the patient should receive at each stage of the recovery process. The final medical discharge location is predicted as a result of sequential predictions of drugs and features. In this work, we combine model ensembling and multi-stage iterative learning for solving an optimal drug prescription generation task as a contribution to the existing literature.

Multi-stage knowledge distillation for sequential recommendation with interest knowledge

Sequential models based on deep learning are widely used in sequential recommendation task, but the increase of model parameters results in a higher latency in the inference stage, which limits the real-time performance of the model. In order to make the model strike a balance between efficiency and effectiveness, the knowledge distillation technology is adopted to transfer the pre-trained knowledge from the large teacher model to the small student model. We propose a multi-stage knowledge distillation method based on interest knowledge, including interest representation knowledge and interest drift knowledge. In the process of knowledge transfer, expert distillation is designed to transform the knowledge dimension of student model to alleviate the loss of original knowledge information. Specially, curriculum learning is introduced for multi-stage knowledge learning, which further makes the teacher model effectively transfer the knowledge to the student model with limited ability. The proposed method on three real-world datasets including MovieLen-1M, Amazon Game and Steam datasets. The experimental results demonstrate that our method is superior to the other compared distillation method significantly and multi-stage learning makes the student model achieve the knowledge step by step for improvement.

A generalized dual-domain generative framework with hierarchical consistency for medical image reconstruction and synthesis

Medical image reconstruction and synthesis are critical for imaging quality, disease diagnosis and treatment. Most of the existing generative models ignore the fact that medical imaging usually occurs in the acquisition domain, which is different from, but associated with, the image domain. Such methods exploit either single-domain or dual-domain information and suffer from inefficient information coupling across domains. Moreover, these models are usually designed specifically and not general enough for different tasks. Here we present a generalized dual-domain generative framework to facilitate the connections within and across domains by elaborately-designed hierarchical consistency constraints. A multi-stage learning strategy is proposed to construct hierarchical constraints effectively and stably. We conducted experiments for representative generative tasks including low-dose PET/CT reconstruction, CT metal artifact reduction, fast MRI reconstruction, and PET/CT synthesis. All these tasks share the same framework and achieve better performance, which validates the effectiveness of our framework. This technology is expected to be applied in clinical imaging to increase diagnosis efficiency and accuracy.

Multistage progressive learning based speech enhancement using time-frequency attentive squeezed temporal convolutional networks

Speech enhancement (SE) is an important method for improving speech quality and intelligibility in noisy environments. An effective speech enhancement model depends on precise modelling of the long-range dependencies of noisy speech. Several recent studies have examined ways to enhance speech by capturing the long-term contextual information. For speech enhancement, the time-frequency (T-F) distribution of speech spectral components is also important, but is usually ignored in these studies. The multi-stage learning method is an effective way to integrate various deep-learning modules at the same time. The benefit of multi-stage training is that the optimization target can be iteratively updated stage by stage. In this paper speech enhancement is investigated by multi-stage learning using a multistage structure in which time-frequency attention (TFA) blocks are followed by stacks of squeezed temporal convolutional networks (S-TCN) with exponentially increasing dilation rates. To reinject original information into later stages, a feature fusion (FF) block is inserted at the input of later stages to reduce the possibility of speech information being lost in the early stages. The S-TCN blocks are responsible for temporal sequence modelling task. The time-frequency attention (TFA) is a simple but effective network module that explicitly exploits position information to generate a 2D attention map to characterise the salient T-F distribution of speech by using two branches, time-frame attention and frequency attention in parallel. A set of utterances from the LibriSpeech and Voicebank databases are used to evaluate the performance of the proposed SE. Extensive experiments have demonstrated that the proposed model consistently improves the performance over existing baselines across two widely used objective metrics such as PESQ and STOI. The average PESQ and STOI for proposed model are boosted by a factor of 41.7% and 5.4% for Libri speech dataset, 36.10% and 3.1% for Voice bank dataset as compared to noisy speech. Additionally, we explored the generalization of the proposed TFA-S-TCN model across different speech datasets through cross data base analysis. A significant improvement in system robustness to noise is also shown by our evaluation results using the TFA module.