Towards understanding memory buffer based continual learning.

DeepONet for solving nonlinear partial differential equations with physics-informed training.

Safety-critical control using high-dimensional sensory feedback from optical data (e.g., images, point clouds) poses significant challenges in domains like autonomous driving and robotic surgery. Control can rely on low-dimensional states estimated from high-dimensional data. However, the estimation errors often follow complex, unknown distributions that standard probabilistic models fail to capture, making formal safety guarantees challenging. In this work, we introduce a novel characterization of these general estimation errors using sub- Gaussian noise with bounded mean. We develop a new technique for uncertainty propagation of proposed noise characterization in linear systems, which combines robust set-based methods with the propagation of sub-Gaussian variance proxies. We further develop a Model Predictive Control (MPC) framework that provides closed-loop safety guarantees for linear systems under the proposed noise assumption. We apply this MPC approach in an ultrasound-image-guided robotic spinal surgery pipeline, which contains deep-learning-based semantic segmentation, image-based registration, high-level optimization-based planning, and low-level robotic control. To validate the pipeline, we developed a realistic simulation environment integrating real human anatomy, robot dynamics, efficient ultrasound simulation, as well as in-vivo data of breathing motion and drilling force. Evaluation results in simulation demonstrate the potential of our approach for solving complex image-guided robotic surgery task while ensuring safety.

Recently, the block-term decomposition with rank-(Lr,Lr,1) (termed as LL1 decomposition), which decomposes a third-order tensor into the sum of the outer products between vector and matrix factors, has received increasing attention for high-dimensional image reconstruction. However, the fixed low-rank matrix decomposition in LL1 is restricted to third-order tensors, which hinders its development for higher-order tensor data (i.e., order N>3). To address this, we propose a Block Customized Topology Term Decomposition (BCTD), which represents an Nth-order tensor as a sum of outer products of basis vectors and customized (N-1)th-order coefficient tensors with flexible internal topological structures. The proposed BCTD enjoys two advantages: Firstly, it allows tackling higher-order tensors beyond the third-order tensor setting of LL1, which can better preserve the high-dimensional structure of the tensor. Secondly, it allows each term to have a customized topological structure beyond the fixed topological structure (i.e., low-rank matrix decomposition) in LL1, which can better explore the intrinsic high-dimensional low-rank structures of the tensor. To evaluate the performance of the proposed BCTD, we build the corresponding high-dimensional image reconstruction model and provide a theoretical generalization error bound between the recovered tensor of the proposed model and the underlying tensor. To solve the resulting optimization problem, we apply a proximal alternating minimization (PAM)-based algorithm with a theoretical convergence guarantee. Extensive experimental results on high-dimensional image completion and compression tasks using real-world datasets (color videos and light field images) demonstrate the superiority of the proposed model over other baseline models.

We consider the transfer learning problem in the linear regression model, where the source domain and target domain have different features and the model exhibits heteroscedasticity. The existing homogeneous transfer learning methods cannot yet handle this type of problem. In this work, a transfer learning algorithm is proposed, which integrates data of varying dimensions and accounts for heteroscedasticity, thereby yielding a data-pooling estimator. The algorithm is simple to implement and easy to operate. The theoretical properties of the proposed estimator are established, including an upper bound on the generalization error and robustness against negative transfer. Simulation studies indicate that the proposed method performs well in terms of parameter estimation. The effectiveness of the proposed method is also validated using real-life datasets in UCI public repository, demonstrating favorable performance compared to the existing methods.

Abstract Understanding the generalization properties of neural networks on simple input–output distributions is key to explaining their performance on real datasets. The classical teacher–student setting, where a network is trained on data generated by a teacher model, provides a canonical theoretical test bed. In this context, a complete theoretical characterization of fully connected one-hidden-layer networks with generic activation functions remains missing. In this work, we develop a general framework for such networks with large width, yet much smaller than the input dimension. Using methods from statistical physics, we derive closed-form expressions for the typical performance of both finite-temperature (Bayesian) and empirical risk minimization estimators in terms of a small number of order parameters. We uncover a transition to a specialization phase, where hidden neurons align with teacher features once the number of samples becomes sufficiently large and proportional to the number of network parameters. Our theory accurately predicts the generalization error of networks trained on regression and classification tasks using either noisy full-batch gradient descent (GD) (Langevin dynamics) or deterministic full-batch GD.

Regression-based multisource conditional domain adaptation for policy outcome prediction.

The increasing proportion of renewable energy leads to frequent changes in turbine load, making the regulating stage more prone to degradation. Traditional anomaly detection methods lack sufficient sensitivity and generalization. To address this issue, this study proposes a method combining multi-attention gated fusion and physical information learning. A gated fusion mechanism is proposed to adaptively extract and fuse key temporal and feature information. Furthermore, the generalization ability of the model is improved by introducing physical constraints derived from the relationship between pressure, temperature, and valve position. Finally, a dynamic temperature prediction model is established using the multi-output long short-term memory neural network. Experiments using actual power plant data demonstrate that the proposed method effectively improves the accuracy of post-regulating-stage temperature prediction and the sensitivity of anomaly detection. The proposed gating fusion method improves prediction accuracy by 4.6% compared to direct addition, while the fusion of physical information reduces the generalization error by more than 6%. In addition, compared to traditional deep learning and machine learning models, the proposed method improves anomaly detection accuracy by at least 3.9%. This research is of great significance for the safe operation of thermal power units and the power grid.

Cross-validation(CV) is widely used for model evaluation and selection in statistical learning. In genomic prediction, linear, mixed-effect, and regularization-based models are commonly applied to predict phenotypic traits from genotype. However, due to linkage disequilibrium(LD), relatedness, and high dimensionality, CV can underestimate true generalization error in genomic settings. In this study, we examine CV bias across seven widely used genomic prediction methods, including ordinary least squares(OLS), generalized least squares(GLS), ridge regression, Lasso, elastic net(ENET), linear mixed models(LMM), and the Bayesian sparse linear mixed model(BSLMM). Using genotypes from the 1000 Genomes Project with simulated phenotypes and an independently generated dataset that removes linkage disequilibrium while preserving allele frequencies, we show that CV consistently produces optimistic error estimates. Building on an existing bias-corrected CV framework that accounts for correlation structure, we derive explicit correction forms for five linear and mixed-effect models with closed-form solutions, and show how the correction can be approximately extended to BSLMM via its equivalence to LMM. The correction substantially reduces CV bias for correctable models, while Lasso and ENET illustrate the theoretical limits of the approach. These results clarify when and why CV is misleading in genomic prediction and provide practical guidance for bias-aware error assessment under genomic dependence.

A bstract In Type II Calabi-Yau string compactifications, S-duality predicts that suitable generating series of BPS indices counting microstates of D4-D2-D0 black holes are in general mock modular forms of higher depth. The non-holomorphic contributions needed to cancel the anomaly under modular transformations involve certain indefinite theta series with kernels constructed from generalized error functions. Physically, these contributions are expected to arise from a spectral asymmetry in the continuum of scattering states of n BPS dyons with mutually non-local charges. For n = 2, the (standard, depth one) error function completion was derived long ago by explicitly computing the bosonic and fermionic density of states in the two-body supersymmetric quantum mechanics. Here we derive the general non-holomorphic completion for an arbitrary number of centers by evaluating the refined Witten index of the supersymmetric quantum mechanics using localization. In a nutshell, the index reduces to an integral over ℝ 3 n − 3 (the relative location of the centers), and splits into an integral over the 2 n – 2 dimensional phase space of BPS ground states times an integral over n – 1 transverse directions, which ultimately produces the expected generalized error functions.

Classical statistical learning theory suggests that models learn data intricacies by fitting a parsimonious model, which can lead to irreducible error components. Overfitting occurs when a model fits the data so well that it becomes too good to be true, and when assessed for unseen data, it performs poorly. The bias-variance trade-off deals with balancing complexity and generalization error. Increasing the number of parameters in models increases the chances of poorly sampling in specific directions, leading to higher variance. This phenomenon is known as benign overfitting. This paper presents a position paper and brief theoretical review that synthesizes key analytical results from the growing literature on benign overfitting. It focuses on overparameterized linear regression analysis, which has two major benefits: decreasing the likelihood of overfitting and uncovering hidden trends in data. However, the generalization bounds for overparameterized models do not explain important empirical observations, and the case of dataset shift remains unexplored in this regime. Rather than proposing a new algorithm or empirical method, the paper aims to clarify conceptual mechanisms underlying benign overfitting and to highlight limitations of current theoretical explanations, particularly under dataset shift. The paper concludes by identifying open theoretical questions relevant to the foundational understanding of modern machine learning systems.

The recent success of large language models (LLMs) has sparked a growing interest in training large-scale models. As the model size continues to scale, concerns are growing about the depletion of high-quality, well-curated training data. This has led practitioners to explore training approaches like Federated Learning (FL), which can leverage the abundant data on edge devices while maintaining privacy. However, the decentralization of training datasets in FL introduces challenges to scaling large models, a topic that remains under-explored. This paper fills this gap and provides qualitative insights on generalizing the previous model scaling experience to federated learning scenarios. Specifically, we derive a PAC-Bayes (Probably Approximately Correct Bayesian) upper bound for the generalization error of models trained with stochastic algorithms in federated settings and quantify the impact of distributed training data on the optimal model size by finding the analytic solution of model size that minimizes this bound. Our theoretical results demonstrate that the optimal model size has a negative power law relationship with the number of clients if the total training compute is unchanged. Besides, we also find that switching to FL with the same training compute will inevitably reduce the upper bound of generalization performance that the model can achieve through training, and that estimating the optimal model size in federated scenarios should depend on the average training compute across clients. Furthermore, we also empirically validate the correctness of our results with extensive training runs on different models, network settings, and datasets.

Zero-shot generalization to out-of-distribution (OOD) teammates and opponents in multi-agent systems (MASs) remains a fundamental challenge for general-purpose AI, especially in open-ended interaction scenarios. Existing multi-agent reinforcement learning (MARL) paradigms, such as self-play and population-based training, often collapse to a limited subset of Nash equilibria, leaving agents brittle when faced with semantically diverse, unseen behaviors. Recent approaches that invoke Large Language Models (LLMs) at run time can improve adaptability but introduce substantial latency and can become less reliable as task horizons grow; in contrast, LLM-assisted reward-shaping methods remain constrained by the inefficiency of the inner reinforcement-learning loop. To address these limitations, we propose LLM-TOC (LLM-Driven Theory-of-Mind Adversarial Curriculum), which casts generalization as a bi-level Stackelberg game: in the inner loop, a MARL agent (the follower) minimizes regret against a fixed population, while in the outer loop, an LLM serves as a semantic oracle that generates executable adversarial or cooperative strategies in a Turing-complete code space to maximize the agent’s regret. To cope with the absence of gradients in discrete code generation, we introduce Gradient Saliency Feedback, which transforms pixel-level value fluctuations into semantically meaningful causal cues to steer the LLM toward targeted strategy synthesis. We further provide motivating theoretical analysis via the PAC-Bayes framework, showing that LLM-TOC converges at rate O(1/K) and yields a tighter generalization error bound than parameter-space exploration under reasonable preconditions. Experiments on the Melting Pot benchmark demonstrate that, with expected cumulative collective return as the core zero-shot generalization metric, LLM-TOC consistently outperforms self-play baselines (IPPO and MAPPO) and the LLM-inference method Hypothetical Minds across all held-out test scenarios, reaching 75% to 85% of the upper-bound performance of Oracle PPO. Meanwhile, with the number of RL environment interaction steps to reach the target relative performance as the core efficiency metric, our framework reduces the total training computational cost by more than 60% compared with mainstream baselines.

Multitask learning with a pretext task has excelled in time-series classification task lacking labeled data. The key to multitask learning is to build a pretext task and learn the most representative feature from the raw time series. In this article, we propose trend and order features for semi-supervised time-series classification via multitask learning (TOFL). Specifically, we propose a simple but effective pretext task-self-sequence order prediction (SOP)-to discover the order relation. In addition, we design a gradual trend fusion (GTF) block concatenating different trend features as the shared backbone network basis element to obtain high-quality trend features for the SOP task. Finally, we not only theoretically analyze the uniform stability and generalization error of TOFL but also evaluate the results compared with state-of-the-art (SOTA) supervised and semi-supervised methods on the 128 UCR datasets and three real-world datasets. TOFL demonstrates a high level of competitiveness and, in most cases, closely matches or even surpasses SOTA methods in terms of accuracy. The source code and data of TOFL are freely available at: https://github.com/Sample-design-alt/TOFL.

ABSTRACT Scientific prediction and evaluation of teaching activities are critical for higher vocational colleges to optimize teaching strategies. However, existing college English teaching evaluation studies lack systematic integration of machine learning models, leading to low prediction accuracy. To address this issue, this study proposes a linear weighted fusion model of Logistic Regression (LR) and Gradient Boosting Decision Tree (GBDT) for English exam pass rate prediction. Based on 14,000 teaching evaluation samples, the model is optimized through SMOTE sampling, L1 regularization feature selection, and weight tuning (optimal weight α = 0.4 for LR). Experimental results show that the fusion model achieves an AUC value of 0.632, which is 5.2–7.2 percentage points higher than single models, with a generalization error rate of only 12%. Further feature importance analysis reveals that question‐type scores, assignment completion rate, and English training frequency are the top three key factors affecting pass rates, while test format and question‐type categories have non‐significant impacts. This study provides a data‐driven solution for college English teaching evaluation, offering actionable insights for personalized instruction and filling the gap of insufficient data support in traditional evaluation methods.

Shallow and ensemble deep randomized neural network for anomaly detection.

Abstract Neural scaling laws-power-law relationships between generalization errors and characteristics of deep learning models-are vital tools for developing reliable models while managing limited resources. Although the success of large language models highlights the importance of these laws, their application to deep regression models remains largely unexplored. Here, we empirically investigate neural scaling laws in deep regression using a parameter estimation model for twisted van der Waals magnets. We observe power-law relationships between the loss and both training dataset size and model capacity across a wide range of values, employing various architectures-including fully connected networks, residual networks, and vision transformers. Furthermore, the scaling exponents governing these relationships range from 1 to 2, with specific values depending on the regressed parameters and model details. The consistent scaling behaviors and their large scaling exponents suggest that the performance of deep regression models can improve substantially with increasing data size.

The looming bias describes systematic differences in the perception of looming as compared to receding stimuli. To date, the most prominent and successful theory put forward to account for this bias is the adaptive bias theory, based on the more general error management theory framework, which argues for a perceptual bias for looming stimuli to err on the side of safety. We challenge this notion by providing evidence using the established probe comparison task from the representational momentum literature, in which the final stimulus configuration is probed. For intensity-changing sounds indicating looming/receding sound sources, no systematic overestimation in intensity change direction for the perceived final sound intensity of looming, approaching stimuli was observed. Across two auditory experiments using either classical sine wave (Experiment 1) or more complex tones (Experiment 2), we replicated the finding of no shift in intensity change direction for looming stimuli, even when accounting for general, change-independent biases. We provide an alternative framework, the speed prior account of motion perception, to explain the present, as well as further, currently unexplained findings in the literature.

Abstract Neural network-based methods have shown great promise in stably solving ill-posed inverse problems. In this work we focus on the inverse problem of recovering point sources, an important class of applied inverse problems. Despite their potential neural network-based methods for identifying point sources remain underdeveloped, primarily due to the inherent singularity of the solution. To address this challenge we develop a novel neural algorithm for identifying point sources, utilizing the singularity enrichment technique. We employ the fundamental solution and neural networks to represent the singular and regular parts, respectively, and then minimize an empirical loss involving the intensities and locations of unknown point sources and the parameters of the neural network. Moreover, by combining the conditional stability argument of the inverse problem with the generalization error of the empirical loss we conduct a rigorous error analysis of the algorithm. We demonstrate the effectiveness of the method with several challenging experiments.

Abstract This paper aims to provide error bounds on Physics-Informed Neural Network (PINN) in solving KdV equations. We prove that a neural network equipped with two hidden layers and the tanh activation function can reduce the partial differential equation (PDE) residuals arbitrarily. The generalization error and training error can be bounded by the number of training points and the width of the aforementioned neural network. Besides, the upper bound of the total error can be controlled by the generalization error. These error bounds offer a theoretical understanding of PINN's ability in solving KdV equations. A series of parameterized KdV equations are also conducted to demonstrate the performance of PINN when solving KdV equations.