Parameter Update Research Articles

RBNN with Cycle-based Parameter Updates and its Application to RUL Prediction

Abstract Bayesian neural networks (BNNs) combine Bayesian theory with deep learning, providing a probabilistic interpretation of deep learning models. Traditional BNNs typically assume that network’s parameters follow a standard Gaussian distribution, which may not accurately reflect the actual situation. To address this issue, this paper proposes a Recurrent Bayesian Neural Network (RBNN) that employs two trainable Gaussian distributions as priors. Furthermore, an alternating cycle model parameter updation algorithm is designed accordingly, which not only prevents overfitting to specific data during training but also enhances the robustness of model parameters in error approximation. Additionally, this RBNN module can be integrated with classical neural network architecture conveniently. By integrating the RBNN with a temporal convolutional network (TCN), the effectiveness of the proposed approach was validated using a dataset for remaining useful life (RUL) prediction of gear and bearing components in civil aircraft. Comparative studies show that the integrated model outperforms traditional methods in terms of RUL prediction and uncertainty quantification, offering superior accuracy and robustness, and the RMSE of the RBNN prediction results is reduced by 17.7% and score is increased by 27.5% compared to BNN.

Biomimetic Active Stereo Camera System with Variable FOV

Inspired by the biological eye movements of fish such as pipefish and sandlances, this paper presents a novel dynamic calibration method specifically for active stereo vision systems to address the challenges of active cameras with varying fields of view (FOVs). By integrating static calibration based on camera rotation angles with dynamic updates of extrinsic parameters, the method leverages relative pose adjustments between the rotation axis and cameras to update extrinsic parameters continuously in real-time. It facilitates epipolar rectification as the FOV changes, and enables precise disparity computation and accurate depth information acquisition. Based on the dynamic calibration method, we develop a two-DOF bionic active camera system including two cameras driven by motors to mimic the movement of biological eyes; this compact system has a large range of visual data. Experimental results show that the calibration method is effective, and achieves high accuracy in extrinsic parameter calculations during FOV adjustments.

Communication‐Efficient Distributed Gradient Descent via Random Projection

ABSTRACTThe gradient descent algorithm is one type of communication‐efficient algorithm in distributed computing as it only requires transmitting the gradient vectors to update parameters. However, as the dimensionality of model parameters increases, even transmitting the gradients alone can result in a significant amount of communication costs. On the other hand, many local computers in the distributed system may suffer from communication bandwidth limitations, so the transmission of the full gradients are practically prohibited. Therefore, reducing the communication cost of gradient descent algorithms, particularly the communication bandwidth, becomes an important issue. To address this problem, we propose a randomly projected gradient descent (RPGD) algorithm. The proposed algorithm consists of three main steps. First, the gradient computed by the local computers is compressed into a low‐dimensional vector using a random matrix. Then, these projected gradients are transmitted to the central computer, aggregated and broadcasted back to the local ones. Finally, the local computers recover the gradients to their original dimensions and update the parameters. By employing random projection, we can reduce the communication bandwidth requirements in distributed computing while we can also provide better privacy protection for local computers. We have provided a theoretical convergence analysis of the RPGD algorithm. Extensive numerical studies have been conducted to demonstrate the finite sample performance of the proposed method.

Super-resolution reconstruction of wind fields with a swin-transformer-based deep learning framework

Deep learning approaches that allow for the rapid simulation of high-resolution atmospheric turbulence are expected by using the super-resolution (SR) technique. Recently, the shifted window attention mechanism in Swin-Transformer offers a significant improvement compared with the vanilla attention mechanism. This method restricts the attention computation to a local neighborhood, reducing the computational load to a linear relationship with sequence length. However, its original architecture is unsuitable for the SR in turbulence due to the mismatch with classification, detection, and segmentation tasks. In this study, the hierarchical structure is redesigned, and a new relative position representing approach is introduced to facilitate the SR procedures of turbulent wind. The channel-shuffled perceptual loss is integrated into the loss function to guide the update of weight parameters. The experimental cases of idealized two-dimensional turbulent flow and realistic boundary layer wind are employed to validate the performance. The results suggest that the proposed approach remarkably outperformed the previous Super-Resolution Convolutional Neural Network, Deep Statistical Downscaling, and Regional Climate Model Emulator in wind vectors. It exhibits lower values than the other three networks whether in terms of point-wise metrics like mean square error, mean absolute error, mean absolute percentage error, or perceptual metrics, including structural similarity index measure and probability density functions. The reconstructed wind vectors closely match the target high-resolution results. This study will help advance the application of shifted window attention mechanisms in wind field SR.

IoT-based home control system using NodeMCU and Firebase

Integrating Internet of Things (IoT) technology in home control systems has led to a significant advancement in the control and monitoring of home appliances and environmental conditions. This study presents the development of an IoT-based home control system that uses the NodeMCU ESP8266 microcontroller to manage several home appliances. This system connects to a Firebase real-time database to store or retrieve data in real-time. The user interacts with the IoT system through a mobile application named My Home, developed using Java programming language, and this mobile application is also interfaced with the Firebase cloud server. The developed IoT system allows users to control home appliances such as lights, sockets, fans, and cookers, and it also provides real-time updates of environmental parameters, such as temperature and humidity, which are measured by a DHT11 sensor. In addition, a PIR motion sensor was integrated into the system to enhance the home's security by detecting intrusion. The system's hardware functionality is based on a written Arduino code, which establishes Wi-Fi connectivity to a predefined network, communicates with the Firebase database, handles appliance control and manages sensor data. This IoT-based home control system shows the potential of integrating microcontrollers with cloud services to create a smart, responsive and user-friendly home control platform. The developed IoT system offers a foundation for advancements, thereby making it a valuable contribution to the growing field of smart home technology.

Transformer-based probabilistic demand forecasting with adaptive online learning

Demand forecasting is crucial for the operation and planning of the energy and power industry. Accurate demand forecasting can assist decision-makers in reducing operational or planning costs while optimizing supply chains and resource utilization. Due to various uncertainties and dynamically changing environments, traditional offline deterministic forecasting methods may face difficulty in forecasting demands accurately and be unable to effectively quantify the uncertainties in forecasts. To adapt to dynamic environmental changes, several online learning methods have been proposed, which require large computational resources. To overcome this limitation, this paper proposes a Transformer-based probabilistic demand forecasting with an adaptive online learning algorithm. More specifically, we design a probabilistic decoder for the Transformer to quantify forecast uncertainty. Furthermore, an adaptive online learning algorithm is employed to selectively update parameters based on an adaptive update strategy, reducing computational burden while allowing for timely adaptation to dynamic environmental changes. The performance of the proposed method is evaluated on multiple benchmark datasets and real electricity demand data from fused magnesium furnace (FMF). The results show that the proposed method exhibits superior accuracy in both deterministic and probabilistic forecasting scenarios. Finally, through ablation experiments, the effectiveness of the proposed adaptive online probabilistic forecasting algorithm is validated.

Violation of auditory regularities is reflected in pupil dynamics

The brain builds and maintains internal models and uses them to make predictions. When predictions are violated, the current model can either be updated or replaced by a new model. The latter is accompanied by pupil dilation responses (PDRs) related to locus coeruleus activity/norepinephrine release (LC-NE). Following earlier research, we investigated PDRs associated with transitions between regular and random patterns of tones in auditory sequences. We presented these sequences to participants and instructed them to find gaps (to maintain attention). Transitions from regular to random patterns induced PDRs, suggesting that an internal model attuned to the regular pattern is reset. Transitions from one regular pattern to another regular pattern also induced PDRs, suggesting that they also led to a model reset. In contrast, transitions from random patterns to regular patterns did not induce PDRs, suggesting a gradual update of model parameters. We modelled these findings, using pupil response functions to show how ongoing PDRs and pupil event rates were sensitive to the trial-by-trial changes in the information content of the auditory sequences. Expanding on previous research, we suggest that PDRs—as biomarkers for LC-NE activation—may indicate the extent of prediction violations.

Shape function method of Tikhonov regularization for vertical bending moment identification based on parameter updating

To accurate identifying vertical bending moment and structural parameters for hull structure, a simultaneous identification technique combined with shape function method (SFM) and sensitivity method was developed. The shape function construction incorporated Tikhonov regularization to enhance the stability of SFM (SFM_Tikhonov). Numerical and experimental studies were conducted. Numerical results showed the effectiveness of SFM_Tikhonov. It achieved a reduction of over 30% in both root mean square error (RMSE) and mean absolute error (MAE) compared to SFM of moving least square fitting (SFM_MLSF). The simultaneous identification method effectively improves the accuracy of moment identification in scenarios of wrong initial parameter estimates. Experimental results show that under constant load condition, the performance of the load identification module based on SFM_Tikhonov lags behind numerical examples, although its parameter update module is unaffected. However, the proposed method remains showing its potential for advancing load identification in structural health monitoring systems.

Data and Model Synergy-Driven Rolling Bearings Remaining Useful Life Prediction Approach Based on Deep Neural Network and Wiener Process

Abstract Various remaining useful life (RUL) prediction methods, encompassing model-based, data-driven, and hybrid methods, have been developed and successfully applied to prognostics and health management for diverse rolling bearing. Hybrid methods that integrate the merits of model-based and data-driven methods have garnered significant attention. However, the effective integration of the two methods to address the randomness in rolling bearing full life cycle processes remains a significant challenge. To overcome the challenge, this paper proposes a data and model synergy-driven RUL prediction framework that includes two data and model synergy strategies. First, a convolutional stacked bidirectional long short-term memory network with temporal attention mechanism is established to construct Health Index (HI). The RUL prediction is achieved based on HI and polynomial model. Second, a three-phase degradation model based on the Wiener process is developed by considering the evolutionary pattern of different degradation phases. Then, two synergy strategies are designed. Strategy 1: HI is adopted as the observation value for online updating of physics degradation model parameters under Bayesian framework, and the RUL prediction results are obtained from the physics degradation model. Strategy 2: The RUL prediction results from the data-driven and physics-based model are weighted linearly combined to improve the overall prediction accuracy. The effectiveness of the proposed model is verified using two bearing full life cycle datasets. The results indicate that the proposed approach can accommodate both short-term and long-term RUL predictions, outperforming state-of-the-art single models.

In-context learning enables multimodal large language models to classify cancer pathology images

Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, where models learn from within prompts, bypassing the need for parameter updates. Yet, in-context learning remains underexplored in medical image analysis. Here, we systematically evaluate the model Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) on cancer image processing with in-context learning on three cancer histopathology tasks of high importance: Classification of tissue subtypes in colorectal cancer, colon polyp subtyping and breast tumor detection in lymph node sections. Our results show that in-context learning is sufficient to match or even outperform specialized neural networks trained for particular tasks, while only requiring a minimal number of samples. In summary, this study demonstrates that large vision language models trained on non-domain specific data can be applied out-of-the box to solve medical image-processing tasks in histopathology. This democratizes access of generalist AI models to medical experts without technical background especially for areas where annotated data is scarce.

Estimation and group-feature selection in sparse mixture-of-experts with diverging number of parameters

Mixture-of-experts provide flexible statistical models for a wide range of regression (supervised learning) problems. Often a large number of covariates (features) are available in many modern applications yet only a small subset of them is useful in explaining a response variable of interest. This calls for a feature selection device. In this paper, we present new group-feature selection and estimation methods for sparse mixture-of-experts models when the number of features can be nearly comparable to the sample size. We prove the consistency of the methods in both parameter estimation and feature selection. We implement the methods using a modified EM algorithm combined with proximal gradient method which results in a convenient closed-form parameter update in the M-step of the algorithm. We examine the finite-sample performance of the methods through simulations, and demonstrate their applications in a real data example on exploring relationships in body measurements.

Research on Sustainable Scheduling of Material-Handling Systems in Mixed-Model Assembly Workshops Based on Deep Reinforcement Learning

In order to dynamically respond to changes in the state of the assembly line and effectively balance the production efficiency and energy consumption of mixed-model assembly, this paper proposes a deep reinforcement learning sustainable scheduling model based on the Deep Q network. According to the particularity of the workshop material-handling system, the action strategy and reward and punishment function are designed, and the neural network structure, parameter update method, and experience pool selection method of the original Deep Q network dual neural network are improved. Prioritized experience replay is adopted to form a real-time scheduling method for workshop material handling based on the Prioritized Experience Replay Deep Q network. The simulation results demonstrate that compared with other scheduling methods, this deep reinforcement learning approach significantly optimizes material-handling scheduling in mixed-flow assembly workshops, effectively reducing handling distance while ensuring timely delivery to the assembly line, ultimately achieving maximum output with sustainable considerations.

Lithium-Ion Battery Health Management and State of Charge (SOC) Estimation Using Adaptive Modelling Techniques

Effective health management and accurate state of charge (SOC) estimation are crucial for the safety and longevity of lithium-ion batteries (LIBs), particularly in electric vehicles. This paper presents a health management system (HMS) that continuously monitors a 4s2p LIB pack’s parameters—current, voltage, and temperature—to mitigate risks such as overcurrent and thermal runaway while ensuring balanced charge distribution between cells. An improved online battery model (IOBM) is developed to enhance SOC estimation accuracy. The system utilises forgetting factor recursive least squares (FFRLS) for real-time parameter updates, an adaptive nonlinear sliding mode observer (ANSMO) for SOC estimation, and a long short-term memory (LSTM) network to dynamically adjust capacity based on operating conditions. Validation using the urban dynamometer driving schedule (UDDS) test demonstrated high accuracy, with the proposed battery model achieving a root mean square error (RMSE) of 12.13 mV and the LSTM achieving an RMSE of 0.0118 Ah. Regular updates to the battery’s current capacity, along with the proposed IOBM, significantly improved SOC estimation performance, maintaining estimation errors within 1.08%.

Adaptive hybrid prediction model for adapting to data distribution shifts in machining quality prediction

Abstract The accuracy of data-driven intelligent prediction for machining quality relies on the training samples. However, in actual applications, the continuous operation of machining equipment leads to gradual distribution shifts between the process data and the training samples for modeling. The shifts result in a degradation in the performance of predictive model, previous studies have often overlooked this issue. To tackle with the intricate problem, this research proposes a real-time model optimization approach. Firstly, a method for detecting machining data distribution shifts based on the two-sample Kolmogorov–Smirnov test is proposed. Then, an adaptive hybrid prediction model (AHPM) capable of real-time optimization is developed. This model consists of a deep neural network (DNN) and a broad learning system (BLS). DNN plays a primary role in prediction within the hybrid model with excellent generalization capability. BLS quickly completes optimization prior to DNN with its unique parameter update mechanism to compensate for prediction loss. Experimental results indicate that AHPM achieves the shortest optimization time while maintaining high accuracy, with post-optimization error reduction rates for mean squared error, mean absolute error, and mean absolute percentage error all exceeding 10%. In the test of application to actual machining cases, accuracy improved by 8.88% compared to traditional methods without optimization.

Monitoring batch suspension polymerization process based on calorimetric-state estimation technique

This work introduces an approach to enhance the real-time monitoring of batch suspension polymerization processes through the integration of real-time calorimetry and state estimation techniques. The challenges inherent in monitoring dynamic and complex processes, particularly in the absence of direct measurements, are effectively addressed through the proposed approach. A dynamic model equation specific to batch operation is employed to assess conversion throughout the process, with timely updates of dynamic parameters in the model equation. A nonlinear high gain cascaded observer with calorimetric measurements estimates the overall heat transfer coefficient and reaction heat. Variations in other parameters such as heat capacity and density are derived from estimated conversion data and updated in the model equation. The study also accounts for the heat loss to the surroundings during isoperibolic batch processes. Validation of the proposed soft sensor model is carried out by comparing the estimated and experimental conversion data by conducting MMA suspension polymerization in a reaction calorimeter. Results demonstrate the potential of calorimetric state estimation as an alternative to direct conversion measurements, offering comprehensive and enhanced monitoring of the batch suspension polymerization process.

Deep transfer learning for microseismic waveforms recognition across geological conditions in TBM tunnels

In deeply buried tunneling projects, geological conditions are often complex and varied. Microseismic monitoring systems are extensively deployed to enhance construction safety. However, when the current geological conditions differ from those present during the signal collection for model training, recognition accuracy tends to decline significantly. Therefore, improving the applicability and stability of microseismic waveform recognition models across varying geological conditions has emerged as a critical challenge. To address this issue, we first analyze the impact of lithological changes and the development of structural planes on the features of microseismic waveforms. Subsequently, we propose a category-domain-aligned transfer learning method that enables the transfer of recognition capabilities across geological conditions by facilitating similar feature extraction and the recognition of cross-geological fracture waveforms. In this model, feature separation modeling enhances the extraction of category features of waveforms under different geological conditions. A deep transfer learning mechanism distinguishes between unique and common features, allowing for the capture of essential features necessary for model parameter updates. Through comparative experiments and feature distribution alignment and visualization, we demonstrate that the accuracy of microseismic waveform recognition across geological conditions achieves 90 %. Additionally, the performance of our method is validated using microseismic signals collected from different sections of the construction site.

Prognosis of reflective pavement cracking development with Bayesian updating and surrogate models

ABSTRACT The main objective of this study is to establish an integrated prognostic method for predicting the development of thermal-induced reflective cracking in asphalt overlay on concrete pavement. The integrated prognosis process for crack growth is developed on the modified Paris’ law and includes three steps: (1) calculation of J-integral; (2) Bayesian updating of crack growth parameters and (3) prediction of remaining life. Finite element modelling (FEM) was conducted to calculate J-integrals at different crack depths and cold weather conditions. A surrogate model was built based on the original Kriging method and it shows high accuracy in predicting J-integrals from maximum air temperature, temperature drop and crack depth. The Bayesian updating process starts from the laboratory-measured distribution of fracture model parameters (A and n) in Paris’ Law and converges to the true values to match the measured crack depths at different loading cycles from field inspection. On the other hand, the consideration of material damage in asphalt overlay avoids over-estimation of remaining loading cycles for reflective cracking to reach the pavement surface. Further study is recommended to accurately assess in-situ pavement modulus and detect crack depth over concrete joints for implementation of the proposed prognostics method.

Overcoming Catastrophic Forgetting in Continual Learning by Exploring Eigenvalues of Hessian Matrix.

Neural networks tend to suffer performance deterioration on previous tasks when they are applied to multiple tasks sequentially without access to previous data. The problem is commonly known as catastrophic forgetting, a significant challenge in continual learning (CL). To overcome the catastrophic forgetting, regularization-based CL methods construct a regularization-based term, which can be considered as the approximation loss function of previous tasks, to penalize the update of parameters. However, the rigorous theoretical analysis of regularization-based methods is limited. Therefore, we theoretically analyze the forgetting and the convergence properties of regularization-based methods. The theoretical results demonstrate that the upper bound of the forgetting has a relationship with the maximum eigenvalue of the Hessian matrix. Hence, to decrease the upper bound of the forgetting, we propose eiGenvalues ExplorAtion Regularization-based (GEAR) method, which explores the geometric properties of the approximation loss of prior tasks regarding the maximum eigenvalue. Extensive experimental results demonstrate that our method mitigates catastrophic forgetting and outperforms existing regularization-based methods.

Developing healthcare language model embedding spaces

Pre-trained Large Language Models (LLMs) have revolutionised Natural Language Processing (NLP) tasks, but often struggle when applied to specialised domains such as healthcare. The traditional approach of pre-training on large datasets followed by task-specific fine-tuning is resource-intensive and poorly aligned with the constraints of many healthcare settings. This presents a significant challenge for deploying LLM-based NLP solutions in medical contexts, where data privacy, computational resources, and domain-specific language pose unique obstacles.This study aims to develop and evaluate efficient methods for adapting smaller LLMs to healthcare-specific datasets and tasks. We seek to identify pre-training approaches that can effectively instil healthcare competency in compact LLMs under tight computational budgets, a crucial capability for responsible and sustainable deployment in local healthcare settings.We explore three specialised pre-training methods to adapt smaller LLMs to different healthcare datasets: traditional Masked Language modelling (MLM), Deep Contrastive Learning for Unsupervised Textual Representations (DeCLUTR), and a novel approach utilising metadata categories from healthcare settings. These methods are assessed across multiple healthcare datasets, with a focus on downstream document classification tasks. We evaluate the performance of the resulting LLMs through classification accuracy and analysis of the derived embedding spaces.Contrastively trained models consistently outperform other approaches on classification tasks, delivering strong performance with limited labelled data and fewer model parameter updates. While our novel metadata-based pre-training does not further improve classifications across datasets, it yields interesting embedding cluster separability. Importantly, all domain-adapted LLMs outperform their publicly available, general-purpose base models, validating the importance of domain specialisation.This research demonstrates the efficacy of specialised pre-training methods in adapting compact LLMs to healthcare tasks, even under resource constraints. We provide guidelines for pre-training specialised healthcare LLMs and motivate continued inquiry into contrastive objectives. Our findings underscore the potential of these approaches for aligning small LLMs with privacy-sensitive medical tasks, offering a path toward more efficient and responsible NLP deployment in healthcare settings. This work contributes to the broader goal of making advanced NLP techniques accessible and effective in specialised domains, particularly where resource limitations and data sensitivity are significant concerns.

Full-scale wind turbine performance assessment using the turbine performance integral (TPI) method: a study of aerodynamic degradation and operational influences

Abstract. This study investigates how blade aerodynamic modifications, including leading edge roughness (LER), influence wind turbine performance over their operational lifespan. It introduces a methodology developed to examine the intricate relationship between blade erosion, blade enhancements, operations and maintenance (O&M) events, control programmable logic controller (PLC) parameter updates, and their cumulative impact on turbine efficiency. Analysing data from 12 multi-megawatt offshore turbines over a 12-year period, the investigation hinges on the integration of supervisory control and data acquisition (SCADA) data, O&M records, and air density corrections. A key contribution is the development of the turbine performance integral (TPI) method, which, for the investigated turbines, leverages generator speed and power output data to track performance trajectories. Seasonal trend decomposition using locally estimated scatterplot smoothing (STL) further isolates long-term trends and seasonal variations in performance. Despite data availability and quality limitations, the study reveals significant findings concerning the impact of manufacturer software updates on turbine control strategies, resulting in improved performance; the variable effects of blade repairs and enhancements; and the complex interaction between O&M events and performance. This work applies a methodical approach and statistical rigour, offering a path forward for effectively monitoring wind turbine efficiency and advancing renewable energy.