Average Model Research Articles

Self-distillation with model averaging

Knowledge distillation (KD) and model averaging (MA) are prominent techniques for enhancing the efficiency and effectiveness of deep neural networks (DNNs). MA produces an average model by averaging multiple checkpoints along the trajectory of the stochastic gradient descent (SGD), which tends to converge toward the flatter side of the local minimum (Fig. 2). However, MA operates offline and does not influence the specific local minimum to which the network converges. By contrast, KD transfers knowledge from a teacher model to a student model, guiding the student toward a better local minimum. Despite this, KD can still result in a poor convergence location within the minimum if the teacher over/under-regularizes the student model's behavior. By combining KD and MA, we aim to leverage KD for a favorable local minimum and MA for a robust convergence location within the local minimum. However, our results revealed that these two techniques were incompatible. This study empirically analyzed the causes of this incompatibility and proposed an annealing knowledge distillation (AKD) scheme to address it. Building on MA and AKD, we introduced a general self-distillation framework referred to as self-distillation with model averaging (SDMA). To the best of our knowledge, SDMA is the first general distillation framework that aims to achieve a favorable local minimum and a robust convergence location. Extensive experiments demonstrated that SDMA significantly improved the generalization performance of DNNs.

On Optimality of Mallows Model Averaging

In the past decades, model averaging (MA) has attracted much attention as it has emerged as an alternative tool to the model selection (MS) statistical approach. Hansen introduced a Mallows model averaging (MMA) method with model weights selected by minimizing a Mallows’ C p criterion. The main theoretical justification for MMA is an asymptotic optimality (AOP), which states that the risk/loss of the resulting MA estimator is asymptotically equivalent to that of the best but infeasible averaged model. MMA’s AOP is proved in the literature by either constraining weights in a special discrete weight set or limiting the number of candidate models. In this work, it is first shown that under these restrictions, however, the optimal risk of MA becomes an unreachable target, and MMA may converge more slowly than MS. In this background, a foundational issue that has not been addressed is: When a suitably large set of candidate models is considered, and the model weights are not harmfully constrained, can the MMA estimator perform asymptotically as well as the optimal convex combination of the candidate models? We answer this question in both nested and non-nested settings. In the nested setting, we provide finite sample inequalities for the risk of MMA and show that without unnatural restrictions on the candidate models, MMA’s AOP holds in a general continuous weight set under certain mild conditions. In the non-nested setting, a sufficient condition and a negative result are established for the achievability of the optimal MA risk. Implications on minimax adaptivity are given as well. The results from simulations and real data analysis back up our theoretical findings. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

PilY1 regulates the dynamic architecture of the type IV pilus machine in Pseudomonas aeruginosa

Type IV pili (T4P) produced by the pathogen Pseudomonas aeruginosa play a pivotal role in adhesion, surface motility, biofilm formation, and infection in humans. Despite the significance of T4P as a potential therapeutic target, key details of their dynamic assembly and underlying molecular mechanisms of pilus extension and retraction remain elusive, primarily due to challenges in isolating intact T4P machines from the bacterial cell envelope. Here, we combine cryo-electron tomography with subtomogram averaging and integrative modelling to resolve in-situ architectural details of the dynamic T4P machine in P. aeruginosa cells. The T4P machine forms 7-fold symmetric cage-like structures anchored in the cell envelope, providing a molecular framework for the rapid exchange of major pilin subunits during pilus extension and retraction. Our data suggest that the T4P adhesin PilY1 forms a champagne-cork-shaped structure, effectively blocking the secretin channel in the outer membrane whereas the minor-pilin complex in the periplasm appears to contact PilY1 via the central pore of the secretin gate. These findings point to a hypothetical model where the interplay between the secretin protein PilQ and the PilY1-minor-pilin priming complex is important for optimizing conformations of the T4P machine in P. aeruginosa, suggesting a gate-keeping mechanism that regulates pilus dynamics.

Time-Aware Missing Healthcare Data Prediction Based on ARIMA Model.

Healthcare uses state-of-the-art technologies (such as wearable devices, blood glucose meters, electrocardiographs), which results in the generation of large amounts of data. Healthcare data is essential in patient management and plays a critical role in transforming healthcare services, medical scheme design, and scientific research. Missing data is a challenging problem in healthcare due to system failure and untimely filing, resulting in inaccurate diagnosis treatment anomalies. Therefore, there is a need to accurately predict and impute missing data as only complete data could provide a scientific and comprehensive basis for patients, doctors, and researchers. However, traditional approaches in this paradigm often neglect the effect of the time factor on forecasting results. This article proposes a time-aware missing healthcare data prediction approach based on the autoregressive integrated moving average (ARIMA) model. We combine a truncated singular value decomposition (SVD) with the ARIMA model to improve the prediction efficiency of the ARIMA model and remove data redundancy and noise. Through the improved ARIMA model, our proposed approach (named MHDP SVD_ARIMA) can capture underlying pattern of healthcare data changes with time and accurately predict missing data. The experiments conducted on the WISDM dataset show that MHDP SVD_ARIMA approach is effective and efficient in predicting missing healthcare data.

A stability improvement control strategy research for LLC resonant converter with single neuron PI-I double closed loop control

LLC resonant converters are widely used in the aerospace field. However, under traditional single-loop control, LLC resonant converters are prone to instability in the face of high-power load disturbances, which fails to meet the high stability requirements of the aerospace field. In order to enhance the stability and disturbance rejection capability of the LLC resonant converter, this paper proposes a stability improvement strategy with additional resonant voltage integral control based on the generalized state-space aver-aged model of the LLC resonant converter. By combining eigenvalue sensitivity analysis and eigenvalue analysis theory, the system stability is further optimized. To achieve control parameter adaptive adjustment, a single neuron algorithm is introduced, resulting in an LLC resonant converter with a single neuron PI-I double closed-loop control. Simulation results indicate that the proposed LLC resonant converter with single neuron PI-I double closed-loop control exhibits improved stability compared to the LLC resonant converter under traditional single-loop control.

Generalized state‐space averaging modeling to fourth‐order power converters operating in DCM

AbstractThis paper presents a novel methodology for correcting the state‐space averaging (SSA) modeling for DC‐DC power converters represented for fourth‐order systems operating in discontinuous conduction mode (DCM). SSA modeling has been widely employed for converters operating in continuous conduction mode (CCM), obtaining a good similarity between the switched converter and the mathematical model. In contrast, when the converters work in DCM, significant deviations between the average and RMS values imply errors in the SSA model. Hence, the proposed generalized SSA modeling methodology provides an accurate model to aid the controllers and system design. Consequently, the proposed technique can be applied to distinct fourth‐order power converters such as Zeta, Ćuk, and SEPIC in several applications. Additionally, SSA corrections can be associated with different techniques for obtaining the small‐signals representation. Simulation and experimental results and mathematical analyses that prove the feasibility and functionality of the proposal are addressed.

Model-based bioequivalence approach for sparse pharmacokinetic bioequivalence studies: Model selection or model averaging?

Conventional pharmacokinetic (PK) bioequivalence (BE) studies aim to compare the rate and extent of drug absorption from a test (T) and reference (R) product using non-compartmental analysis (NCA) and the two one-sided test (TOST). Recently published regulatory guidance recommends alternative model-based (MB) approaches for BE assessment when NCA is challenging, as for long-acting injectables and products which require sparse PK sampling. However, our previous research on MB-TOST approaches showed that model misspecification can lead to inflated type I error. The objective of this research was to compare the performance of model selection (MS) on R product arm data and model averaging (MA) from a pool of candidate structural PK models in MBBE studies with sparse sampling. Our simulation study was inspired by a real case BE study using a two-way crossover design. PK data were simulated using three structural models under the null hypothesis and one model under the alternative hypothesis. MB-TOST was applied either using each of the five candidate models or following MS and MA with or without the simulated model in the pool. Assuming T and R have the same PK model, our simulation shows that following MS and MA, MB-TOST controls type I error rates at or below 0.05 and attains similar or even higher power than when using the simulated model. Thus, we propose to use MS prior to MB-TOST for BE studies with sparse PK sampling and to consider MA when candidate models have similar Akaike information criterion.

Small-signal modeling comparison of dual active bridge converter

The dual active bridge (DAB) converter is a versatile DC-DC converter, which has AC components in the high-frequency transformer within the converter topology as well as DC components at the input and output sides. These AC components are constructed from infinite sinusoids of multiples of the converter's switching frequency. Different approaches have been proposed to model the DAB converter. The generalized averaging modeling (GAM) includes only the first order harmonic of the inductor current with the DC components of the input and output voltages and currents. For better model accuracy, harmonic-domain models have been employed, where the model includes all the harmonics of all the model state variables up to an arbitrary harmonic index. While the harmonic model provides a systematic approach to include the harmonics, this increases the model complexity. To avoid this complexity, the flexible harmonic model is used where harmonics can be selected per each state variable without needing to be in successive order. The aforementioned modeling approaches have been implemented on the DAB converter. This paper aims to compare between the dynamic performance of GAM and the harmonic modeling approaches under different disturbances, where the GAM approach provided a simple model with acceptable results that can be used for controller design, while the LTP model provided a more complex model with better accuracy that is suitable for harmonic studies. The models have been validated and compared using simulations executed on MATLAB /Simulink environment.

Comparison of time series temperature prediction with auto-regressive integrated moving average and recurrent neural network

The region of Beni Mellal, Morocco is heavily dependent on the agricultural sector as its primary source of income. Accurate temperature prediction in agriculture has many benefits including improved crop planning, reduced crop damage, optimized irrigation systems and more sustainable agricultural practices. By having a better understanding of the expected temperature patterns, farmers can make informed decisions on planting schedules, protect crops from extreme temperature events, and use resources more efficiently. The lack of data-driven studies in agriculture impedes the digitalization of farming and the advancement of accurate long-term temperature prediction models. This underscores the significance of research to identify the optimal machine learning models for that purpose. A 22-year time series dataset (2000-2022) is used in the study. The machine-learning model auto-regressive integrated moving average (ARIMA) and deep learning models simple recurrent neural network (SimpleRNN), gated recurrent unit (GRU), and long short-term memory (LSTM) were applied to the time series. The results are evaluated based on the mean absolute error (MAE). The findings indicate that the deep learning models outperformed the machine-learning model, with the GRU model achieving the lowest MAE.

DC Arc Model Superimposing Current-Controlled Arc Noise on a Heuristic Average Model

DC Arc Model Superimposing Current-Controlled Arc Noise on a Heuristic Average Model

Trends in opioid toxicities among people with and without opioid use disorder and the impact of the COVID-19 pandemic in Ontario, Canada: A population-based analysis

BackgroundAcross Canada, the COVID-19 pandemic occurred amidst an ongoing drug toxicity crisis. Although elevated rates of substance-related harms have been observed nationally, it remains unknown if the pandemic state of emergency led to disproportionate increases in opioid toxicities among people with opioid use disorder (OUD) compared to those without. MethodsWe conducted a population-based repeated cross-sectional time series analysis of fatal and non-fatal opioid toxicities between January 1, 2014, and December 31, 2021, in Ontario, Canada. We used interventional autoregressive integrated moving average models to examine the impact of the pandemic on monthly rates of opioid toxicities per 100,000 Ontario residents stratified by people with and without OUD. ResultsWe identified 80,296 opioid toxicities of which 53.5 % occurred among people with OUD. Among 52,052 unique individuals, 60.5 % were male and 46.2 % were 25–44 years old. Between January 2014 and December 2021, the rate of opioid toxicities increased from 2.6 to 10.5 per 100,000 (rate ratio [RR]=4.07). The magnitude of this increase differed among people with OUD (0.8 to 7.4 per 100,000; RR=9.35) and without OUD (1.8 to 3.1 per 100,000; RR=1.74). We observed a significant ramp increase in the overall rate of opioid toxicities following the declaration of the pandemic emergency in March 2020 (+0.19 per 100,000 monthly, 95 % CI: 0.029, 0.36, p = 0.021). In a stratified analysis, we found a similar ramp increase among people with OUD (+0.19 per 100,000 monthly, 95 % CI: 0.10, 0.28, p < 0.001); however, this was not observed among people without OUD (p = 0.95). ConclusionsThe rate of opioid toxicities accelerated across Ontario following the pandemic-related state of emergency, with the majority of this increase among people with OUD. The important differences observed among people with OUD compared with those without, highlights the critical need for improved access to harm reduction and treatment interventions among this population.

Modelling and optimizing microgrid systems with the utilization of real-time residential data: a case study for Palapye, Botswana

Microgrids are becoming a realistic choice for residential buildings due to the increasing need for affordable and sustainable energy solutions in developing nations. Through modeling and simulation, the main goal is to evaluate the viability and performance of a solar microgrid system. Residential load modeling is used, which is vital to developing an effective Energy Management System (EMS) for the microgrid. A residential household’s load metering data is examined using statistical methods, including time series and regression analysis. For the residential community load in this research, Proportional-Integral-Derivative (PID) controllers and Fuzzy Logic Controllers (FLC) are used to generate the necessary Direct Current (DC) microgrid voltage. The simulation research shows that FLC have benefits over PID controllers. The FLC technique performs better at reducing total harmonic distortion, which improves the microgrid system’s overall power quality. The Seasonal Autoregressive Integrated Moving Average (SARIMA) model was found to be the most appropriate and reliable model for the dataset after the performance of the models was evaluated using the metrics. The optimization results also showed that FLC optimization improves the microgrid system’s stability. The exponential Gaussian process regression (GPR) produced the highest R-squared measure of 0.49 and RSME measure of 7.9646, making it the best goodness fit for modeling the total daily energy usage and the peak daily usage.

Tackling Uncertainty: Forecasting the Energy Consumption and Demand of an Electric Arc Furnace with Limited Knowledge on Process Parameters

The iron and steel industry significantly contributes to global energy use and greenhouse gas emissions. The rising deployment of volatile renewables and the resultant need for flexibility, coupled with specific challenges in electric steelmaking (e.g., operation optimization, optimized power purchasing, effective grid capacity monitoring), require accurate energy consumption and demand forecasts for electric steel mills to align with the energy transition. This study investigates diverse approaches to forecast the energy consumption and demand of an electric arc furnace—one of the largest consumers on the grid—considering various forecast horizons and objectives with limited knowledge on process parameters. The results are evaluated for accuracy, robustness, and costs. Two grid connection capacity monitoring approaches—a one-step and a multi-step Long Short-Term Memory neural network—are assessed for intra-hour energy demand forecasts. The one-step approach effectively models energy demand, while the multi-step approach encounters challenges in representing different operational phases of the furnace. By employing a combined statistic–stochastic model integrating a Seasonal Auto-Regressive Moving Average model and Markov chains, the study extends the forecast horizon for optimized day-ahead electricity procurement. However, the accuracy decreases as the forecast horizon lengthens. Nevertheless, the day-ahead forecast provides substantial benefits, including reduced energy balancing needs and potential cost savings.

An Interpretable and Efficient Infinite-Order Vector Autoregressive Model for High-Dimensional Time Series

As a special infinite-order vector autoregressive (VAR) model, the vector autoregressive moving average (VARMA) model can capture much richer temporal patterns than the widely used finite-order VAR model. However, its practicality has long been hindered by its non-identifiability, computational intractability, and difficulty of interpretation, especially for high-dimensional time series. This article proposes a novel sparse infinite-order VAR model for high-dimensional time series, which avoids all above drawbacks while inheriting essential temporal patterns of the VARMA model. As another attractive feature, the temporal and cross-sectional structures of the VARMA-type dynamics captured by this model can be interpreted separately, since they are characterized by different sets of parameters. This separation naturally motivates the sparsity assumption on the parameters determining the cross-sectional dependence. As a result, greater statistical efficiency and interpretability can be achieved with little loss of temporal information. We introduce two l 1 -regularized estimation methods for the proposed model, which can be efficiently implemented via block coordinate descent algorithms, and derive the corresponding nonasymptotic error bounds. A consistent model order selection method based on the Bayesian information criteria is also developed. The merit of the proposed approach is supported by simulation studies and a real-world macroeconomic data analysis. Supplementary materials for this article are available online including a standardized description of the materials available for reproducing the work.

Forecasting weekly dengue incidence in Sri Lanka: Modified Autoregressive Integrated Moving Average modeling approach.

Dengue poses a significant and multifaceted public health challenge in Sri Lanka, encompassing both preventive and curative aspects. Accurate dengue incidence forecasting is pivotal for effective surveillance and disease control. To address this, we developed an Autoregressive Integrated Moving Average (ARIMA) model tailored for predicting weekly dengue cases in the Colombo district. The modeling process drew on comprehensive weekly dengue fever data from the Weekly Epidemiological Reports (WER), spanning January 2015 to August 2020. Following rigorous model selection, the ARIMA (2,1,0) model, augmented with an autoregressive component (AR) of order 16, emerged as the best-fitted model. It underwent initial calibration and fine-tuning using data from January 2015 to August 2020, and was validated against independent 2000 data. Selection criteria included parameter significance, the Akaike Information Criterion (AIC), and Schwarz Bayesian Information Criterion (SBIC). Importantly, the residuals of the ARIMA model conformed to the assumptions of randomness, constant variance, and normality affirming its suitability. The forecasts closely matched observed dengue incidence, offering a valuable tool for public health decision-makers. However, an increased percentage error was noted in late 2020, likely attributed to factors including potential underreporting due to COVID-19-related disruptions amid rising dengue cases. This research contributes to the critical task of managing dengue outbreaks and underscores the dynamic challenges posed by external influences on disease surveillance.

A hybrid model based on STL with simple exponential smoothing and ARMA for wind forecast in a Brazilian nuclear power plant site

In the event of a nuclear accident with release of radioactive effluents into the environment, the Environmental Control System (ECS) at the Central Nuclear Almirante Álvaro Alberto (CNAAA) nuclear complex estimates the movement of the radioactive plume produced and provides forecast for one and two hours ahead. The prediction is based on the current accident and atmospheric data collected from meteorological stations spread across the site. In this way, the ECS estimates the spatial dose and dose rates distribution at ground level. The one- and two-hour forecast made by the ECS use the naïve method, which repeats the last known value. Its results are satisfactory if the next value to be collected does not differ substantially from the last known value. However, observing real data, acquired in loco, this fact does not occur in practice, which can make the forecast considerably erroneous.The purpose of this work is to investigate new approaches to improve ECS forecasting method, making it more accurate in real situations. To this end, the Seasonal and Trend decomposition using Loess (STL) was used together with the Simple Exponential Smoothing (SES) method and the Autoregressive Moving Average (ARMA) model to forecast weather parameters at the Angra site. The STL method was chosen since it has a robust mathematical decomposition method, capable of satisfactorily dealing with time series that have a high frequency of acquisition, the so-called intraday series. This model uses the Loess method, which is a smoothing technique based on locally adjusted regressions. The SES method and ARMA model are the techniques that most adequately fit the trend and remainder components generated by the STL decomposition, respectively. In the scope of this work, the method is applied to forecast wind speed measurements made at one of the meteorological towers used by the ECS. Using the hybrid model developed, an increase the of 101.39% in the Mean Absolute Percentage Error (MAPE), was achieved when compared to the ECS forecast method.

Analysis of China's carbon market price fluctuation and international carbon credit financing mechanism using random forest model.

This study aims to investigate the price changes in the carbon trading market and the development of international carbon credits in-depth. To achieve this goal, operational principles of the international carbon credit financing mechanism are considered, and time series models were employed to forecast carbon trading prices. Specifically, an ARIMA(1,1,1)-GARCH(1,1) model, which combines the Generalized Autoregressive Conditional Heteroskedasticity (GARCH) and Autoregressive Integrated Moving Average (ARIMA) models, is established. Additionally, a multivariate dynamic regression Autoregressive Integrated Moving Average with Exogenous Inputs (ARIMAX) model is utilized. In tandem with the modeling, a data index system is developed, encompassing various factors that influence carbon market trading prices. The random forest algorithm is then applied for feature selection, effectively identifying features with high scores and eliminating low-score features. The research findings reveal that the ARIMAX Least Absolute Shrinkage and Selection Operator (LASSO) model exhibits high forecasting accuracy for time series data. The model's Mean Squared Error, Root Mean Squared Error, and Mean Absolute Error are reported as 0.022, 0.1344, and 0.1543, respectively, approaching zero and surpassing other evaluation models in predictive accuracy. The goodness of fit for the national carbon market price forecasting model is calculated as 0.9567, indicating that the selected features strongly explain the trading prices of the carbon emission rights market. This study introduces innovation by conducting a comprehensive analysis of multi-dimensional data and leveraging the random forest model to explore non-linear relationships among data. This approach offers a novel solution for investigating the complex relationship between the carbon market and the carbon credit financing mechanism.

Frequency prediction of a post-disturbance power system using a hybrid ARIMA and DBN model

The online prediction of power system dynamic frequency helps to guide the choice of control measures quickly and accurately after a disturbance, and this then ensures the reliable and stable operations of a power system. However, the prediction performance of the traditional single model is not accurate enough, and the prediction method cannot reflect the dynamic mechanism of the power system. To address these challenges, based on the analysis of the mechanism of the dynamic operation of a power system, a dynamic frequency online prediction method using the autoregressive integrated moving average (ARIMA) model and the deep belief network (DBN) is proposed in this paper. First, according to the mechanism of the dynamic operation of a power system, the dynamic frequency can be regarded as having two stages after the disturbance occurs. In the first stage, the frequency changes monotonously in the short term, which is predicted by the ARIMA model. Furthermore, the second stage is an oscillation phase with changing amplitude, which is predicted by the DBN. The calibration process is used to combine the two predicted results. Second, the three metrics including the frequency nadir (fnadir), the quasi-steady state frequency (fss), and the frequency curve obtained through the prediction are analyzed to measure the accuracy of the prediction results. Finally, to verify the accuracy of the proposed model, the IEEE 10-generator 39-bus benchmark system is used for verification.

Least Absolute Deviations Estimation for an Uncertain Moving Average Model

The existing parameter estimation methods of the uncertain moving average model are the least square method and ridge estimation. However, for some data with outliers, the least square method cannot eliminate the influence of outliers very well. In order to solve this problem, this paper discusses how to use the parameter estimation method of least absolute deviation to solve the uncertain moving average model, and calculate the mean and variance of the residuals, so as to find the confidence interval of the predicted value. Finally, two examples are given to demonstrate that the uncertain moving average model estimated by the least absolute deviation can predict closer values to the actual values than other estimation methods.

Neural network and machine learning use cases: Indian bond market predictions

This study examines machine learning techniques to investigate how artificial intelligence (AI) affects predicting future trends in the bond market. The bond market offers a global perspective on capital costs for a business by establishing the fair value of the bond issue, which is based on multiple factors. The asset price market, which has employed machine learning (ML) and deep learning (DL) techniques to address the primary forecasting difficulty, surprisingly plays a significant role in predicting future bond market returns. As an outcome, if this gap can be forecast, it can act as the bond market's data-driven long-term direction and yield additional profits. Daily security-specific data for the 10-to-3-year Indian Treasury Bond (ITB) was gathered from 2013 to 2022 and is available in the global government bonds database. The researchers looked at how well the auto-regressive integrated moving average (ARIMA), linear regression, and deep recurrent neural network-long short-term memory (DLSTM) models could predict bond yields and returns in future bond markets. The empirical results demonstrate that the DLSTM models most fairly predict the price of government bonds over both the short and longer horizons when compared to ARIMA and linear regression.