Learning Bias Research Articles

Multidimensional Quantum Generative Modeling by Quantum Hartley Transform

AbstractAn approach for building quantum models based on the exponentially growing orthonormal basis of Hartley kernel functions is developed. First, a differentiable Hartley feature map parameterized by real‐valued argument that enables quantum models suitable for solving stochastic differential equations and regression problems is designed. Unlike the naturally complex Fourier encoding, the proposed Hartley feature map circuit leads to quantum states with real‐valued amplitudes, introducing an inductive bias and natural regularization. Next, a quantum Hartley transform circuit is proposed as a map between computational and Hartley basis. The developed paradigm is applied to generative modeling from solutions of stochastic differential equations, and utilize the quantum Hartley transform for fine sampling from parameterized distributions through an extended register. Finally, the capability of multivariate quantum generative modeling is demonstrated for both correlated and uncorrelated distributions. As a result, the developed quantum Hartley‐based generative models (QHGMs) offer a distinct quantum approach to generative AI at increasing scale.

DESENet: a bilateral network with detail-enhanced semantic encoder for real-time semantic segmentation

Abstract Real-time semantic segmentation is widely used in various domains such as autonomous driving. Most real-time semantic segmentation networks adopt an encoder-decoder structure. During encoding, feature maps undergo multiple downsampling stages to enhance computational efficiency and enlarge the receptive field. However, this process may lead to the loss of detailed information, which may not be fully recovered during upsampling. Moreover, upsampling semantic features may cause blurring due to the lack of spatial detail information. To mitigate these issues, we use spatially guided cross-resolution self-attention to improve the upsampling of the semantic features by supplementing detailed information from the spatial branch. Furthermore, we incorporate an inductive bias into the cross-resolution attention mechanism to enhance its ability to learn generalized features. Additionally, we design a semantic feature extraction block, and a spatial feature extraction branch to construct a lightweight backbone. The results on Cityscapes and CamVid show that the proposed model achieves a good balance between accuracy and efficiency. Specifically, it obtains 74.2 and 71.5 mIoU on the two test datasets with 1.31M parameters, respectively. Code is available on https://github.com/clearwater753/DECENet.

SSDSNet: A Dual-Stream State-Space Network for Efficient Image Denoising with Long-Range Dependency Modeling

Abstract In recent years, significant progress has been made in image denoising, driven by advancements in modern deep neural networks such as Convolutional Neural Networks
(CNNs) and Transformers. However, existing denoising models often face limitations due to inherent local inductive biases or the quadratic computational complexity of their architectures. Recently, Selective Structured State Space Models, exemplified by Mamba, have shown considerable potential in modeling long-range dependencies with linear complexity. Despite this promise, their application in low-level computer vision tasks remains underexplored. In this study, we introduce a straightforward yet effective model, the Dual-Stream State-Space Module (DSSM), specifically designed for image denoising. The DSSM serves as the core component of our model, combining convolutional operations with the
Mamba model to enhance the capability of capturing both local features and global context.
This dual-stream approach leverages the inherent structure of local image patches while effectively modeling long-range dependencies, resulting in feature representations tailored for denoising tasks. Extensive experiments demonstrate the effectiveness of our method. For instance, on the SIDD dataset, our model achieves performance on par with state-of-the-art methods, maintaining a global receptive field while operating with similar computational costs.

Compositional pretraining improves computational efficiency and matches animal behavior on complex tasks.

1Recurrent neural networks (RNN) are ubiquitously used in neuroscience to capture both neural dynamics and behaviors of living systems. However, when it comes to complex cognitive tasks, training RNNs with traditional methods can prove difficult and fall short of capturing crucial aspects of animal behavior. Here we propose a principled approach for identifying and incorporating compositional tasks as part of RNN training. Taking as target a temporal wagering task previously studied in rats, we design a pretraining curriculum of simpler cognitive tasks that reflect relevant sub-computations. We show that this pretraining substantially improves learning efficacy and is critical for RNNs to adopt similar strategies as rats, including long-timescale inference of latent states, which conventional pretraining approaches fail to capture. Mechanistically, our pretraining supports the development of slow dynamical systems features needed for implementing both inference and value-based decision making. Overall, our approach is an important step for endowing RNNs with relevant inductive biases, which is important when modeling complex behaviors that rely on multiple cognitive computations.

Knowledge-based inductive bias and domain adaptation for cell type annotation.

Measurement techniques often result in domain gaps among batches of cellular data from a specific modality. The effectiveness of cross-batch annotation methods is influenced by inductive bias, which refers to a set of assumptions that describe the behavior of model predictions. Different annotation methods possess distinct inductive biases, leading to varying degrees of generalizability and interpretability. Given that certain cell types exhibit unique functional patterns, we hypothesize that the inductive biases of cell annotation methods should align with these biological patterns to produce meaningful predictions. In this study, we propose KIDA, Knowledge-based Inductive bias and Domain Adaptation. The knowledge-based inductive bias constrains the prediction rules learned from the reference dataset, composed of multiple batches, to functional patterns relevant to biology, thereby enhancing the generalization of the model to unseen batches. Since the query dataset also contains gaps from multiple batches, KIDA's domain adaptation employs pseudo labels for self-knowledge distillation, effectively narrowing the distribution gap between model predictions and the query dataset. Benchmark experiments demonstrate that KIDA is capable of achieving accurate cross-batch cell type annotation.

Quantifying Shape and Texture Biases for Enhancing Transfer Learning in Convolutional Neural Networks

Neural networks have inductive biases owing to the assumptions associated with the selected learning algorithm, datasets, and network structure. Specifically, convolutional neural networks (CNNs) are known for their tendency to exhibit textural biases. This bias is closely related to image classification accuracy. Aligning the model’s bias with the dataset’s bias can significantly enhance performance in transfer learning, leading to more efficient learning. This study aims to quantitatively demonstrate that increasing shape bias within the network by varying kernel sizes and dilation rates improves accuracy on shape-dominant data and enables efficient learning with less data. Furthermore, we propose a novel method for quantitatively evaluating the balance between texture bias and shape bias. This method enables efficient learning by aligning the biases of the transfer learning dataset with those of the model. Systematically adjusting these biases allows CNNs to better fit data with specific biases. Compared to the original model, an accuracy improvement of up to 9.9% was observed. Our findings underscore the critical role of bias adjustment in CNN design, contributing to developing more efficient and effective image classification models.

Raising awareness of potential biases in medical machine learning: Experience from a Datathon.

To challenge clinicians and informaticians to learn about potential sources of bias in medical machine learning models through investigation of data and predictions from an open-source severity of illness score. Over a two-day period (total elapsed time approximately 28 hours), we conducted a datathon that challenged interdisciplinary teams to investigate potential sources of bias in the Global Open Source Severity of Illness Score. Teams were invited to develop hypotheses, to use tools of their choosing to identify potential sources of bias, and to provide a final report. Five teams participated, three of which included both informaticians and clinicians. Most (4/5) used Python for analyses, the remaining team used R. Common analysis themes included relationship of the GOSSIS-1 prediction score with demographics and care related variables; relationships between demographics and outcomes; calibration and factors related to the context of care; and the impact of missingness. Representativeness of the population, differences in calibration and model performance among groups, and differences in performance across hospital settings were identified as possible sources of bias. Datathons are a promising approach for challenging developers and users to explore questions relating to unrecognized biases in medical machine learning algorithms. Disadvantaged groups are at risk of being adversely impacted by biased medical machine learning models. To avoid these undesirable outcomes, developers and users must understand the challenges involved in identifying potential biases. We conducted a datathon aimed at challenging a diverse group of participants to explore an open-source patient severity model for potential biases. Five groups of clinicians and informaticians used tools of their choosing to evaluate possible sources of biases, applying a range of analytic techniques and exploring multiple features. By engaging diverse participants with hands-on data experience with meaningful data, datathons have the potential to raise awareness of potential biases and promote best practices in developing fair and equitable medical machine learning models.

CIDGMed: Causal Inference-Driven Medication Recommendation with enhanced dual-granularity learning

Medication recommendation aims to integrate patients’ long-term health records to provide accurate and safe medication combinations for specific health states. Existing methods often fail to deeply explore the true causal relationships between diseases/procedures and medications, resulting in biased recommendations. Additionally, in medication representation learning, the relationships between information at different granularities of medications—coarse-grained (medication itself) and fine-grained (molecular level)—are not effectively integrated, leading to biases in representation learning. To address these limitations, we propose the Causal Inference-driven Dual-Granularity Medication Recommendation method (CIDGMed). Our approach leverages causal inference to uncover the relationships between diseases/procedures and medications, thereby enhancing the rationality and interpretability of recommendations. By integrating coarse-grained medication effects with fine-grained molecular structure information, CIDGMed provides a comprehensive representation of medications. Additionally, we employ a bias correction model during the prediction phase to further refine recommendations, ensuring both accuracy and safety. Through extensive experiments, CIDGMed significantly outperforms current state-of-the-art models across multiple metrics, achieving a 2.54% increase in accuracy, a 3.65% reduction in side effects, and a 39.42% improvement in time efficiency. Additionally, we demonstrate the rationale of CIDGMed through a case study.

On the shortcut learning in multilingual neural machine translation

In this study, we revisit the commonly-cited off-target issue in multilingual neural machine translation (MNMT). By carefully designing experiments on different MNMT scenarios and models, we attribute the off-target issue to the overfitting of the shortcuts of (non-centric, centric) language mappings. Specifically, the learned shortcuts biases MNMT to mistakenly translate non-centric languages into the centric language instead of the expected non-centric language for zero-shot translation. Analyses on learning dynamics show that the shortcut learning generally occurs in the later stage of model training, and multilingual pretraining accelerates and aggravates the shortcut learning. Based on these observations, we propose a simple and effective training strategy to eliminate the shortcuts in MNMT models by leveraging the forgetting nature of model training. The only difference from the standard training is that we remove the training instances that may induce the shortcut learning in the later stage of model training. Without introducing any additional data and computational costs, our approach can consistently and significantly improve the zero-shot translation performance by alleviating the shortcut learning for different MNMT models and benchmarks.

POLYCORE: Polygon-based contour refinement for improved Intravascular Ultrasound Segmentation

Segmentation of the coronary vessel wall in intravascular ultrasound is a fundamental step in guiding coronary intervention. However, it is an challenging task, even for highly skilled cardiologists, due to image artefacts and shadowed regions caused by calcified plaque, guide wires and vessel side branches. Recently, dense-based neural networks have been applied to this task, however, they often fail to predict anatomically plausible contours in these low-signal areas. We propose a novel methodology called Polygon-based Contour Refiner (POLYCORE) that addresses topological error in dense-based segmentation networks using a relational inductive bias through higher-order connections between vertices to learn anatomically rational contours. Our approach remedies the over-smoothing phenomena common in polygon networks by introducing a new vector field refinement module which enables pixel-level detail to be added in an iterative process. POLYCORE is enhanced with augmented polygon aggregation which we show is more effective than typical dense-based test-time augmentation strategies. We achieve state-of-the-art results on two diverse datasets, observing particular improvements when segmenting the lumen structure and in topologically-challenging regions containing shadow artefacts. Our source code is available here: http://orcid.org/https://github.com/kitbransby/POLYCORE.

Topological embedding and directional feature importance in ensemble classifiers for multi-class classification

Cancer is the second leading cause of disease-related death worldwide, and machine learning-based identification of novel biomarkers is crucial for improving early detection and treatment of various cancers. A key challenge in applying machine learning to high-dimensional data is deriving important features in an interpretable manner to provide meaningful insights into the underlying biological mechanismsWe developed a class-based directional feature importance (CLIFI) metric for decision tree methods and demonstrated its use for the The Cancer Genome Atlas proteomics data. The CLIFI metric was incorporated into four algorithms, Random Forest (RF), LAtent VAriable Stochastic Ensemble of Trees (LAVASET), and Gradient Boosted Decision Trees (GBDTs), and a new extension incorporating the LAVA step into GBDTs (LAVABOOST). Both LAVA methods incorporate topological information from protein interactions into the decision function.The different models' performance in classifying 28 cancers resulted in F1-scores of 92.6% (RF), 92.0% (LAVASET), 89.3% (LAVABOOST) and 85.7% (GBDT), with no method outperforming all others for individual cancer type prediction. The CLIFI metric enables visualisation of the model's decision-making functions. The resulting CLIFI value distributions indicated heterogeneity in the expression of several proteins (MYH11, ERα, BCL2) across different cancer types (including brain glioma, breast, kidney, thyroid and prostate cancer) aligning with the original raw expression data.In conclusion, we have developed an integrated, directional feature importance metric for multi-class decision tree-based classification models that facilitates interpretable feature importance assessment. The CLIFI metric can be combined with incorporating topological information into the decision functions of models to introduce inductive bias, enhancing interpretability.

Adaptive scheduling for Internet of Vehicles using deconfounded graph transfer learning

With the rapid development of Internet of Things technology, the Internet of Vehicles has emerged as a significant application. The Internet of Vehicles connects vehicles with traffic infrastructure and other vehicles, enabling real-time information sharing and vehicle dispatching. However, in the Internet of Vehicles environment, traffic scheduling systems face challenges such as sparse raw data and environmental confounding bias, which limit the effectiveness of traditional graph-based reinforcement learning methods. In this paper, we propose a Cross-City Federated Continual Learning Framework for Spatiotemporal Graph Transfer Learning called CCFTL, which removes confounding effects on reinforcement learning and enhances few-shot transfer learning in data-scarce cities. Our approach aims to significantly improve the adaptability and scalability of cross-city vehicle networking technologies in ever-changing environments. This study acts as an innovative research of the deconfounding strategy with spatiotemporal graph meta-knowledge learning, which enables optimization of cross-city meta-knowledge transfer through a federated continual learning approach. This approach effectively reduces learning bias caused by regional differences, which enhances the model’s generalization and adaptability to complex, heterogeneous urban scenarios. Experimental results show that our framework significantly outperforms traditional methods, especially in improving the efficiency and accuracy of traffic dispatch systems in Internet of Vehicles environments with scarce data and the presence of confounding factors.

A large-scale examination of inductive biases shaping high-level visual representation in brains and machines

The rapid release of high-performing computer vision models offers new potential to study the impact of different inductive biases on the emergent brain alignment of learned representations. Here, we perform controlled comparisons among a curated set of 224 diverse models to test the impact of specific model properties on visual brain predictivity – a process requiring over 1.8 billion regressions and 50.3 thousand representational similarity analyses. We find that models with qualitatively different architectures (e.g. CNNs versus Transformers) and task objectives (e.g. purely visual contrastive learning versus vision- language alignment) achieve near equivalent brain predictivity, when other factors are held constant. Instead, variation across visual training diets yields the largest, most consistent effect on brain predictivity. Many models achieve similarly high brain predictivity, despite clear variation in their underlying representations – suggesting that standard methods used to link models to brains may be too flexible. Broadly, these findings challenge common assumptions about the factors underlying emergent brain alignment, and outline how we can leverage controlled model comparison to probe the common computational principles underlying biological and artificial visual systems.

Kernels, data & physics

Abstract Lecture notes from the course given by Professor Julia Kempe at the summer school ‘Statistical physics of Machine Learning’ in Les Houches. The notes discuss the so-called NTK approach to problems in machine learning, which consists of gaining an understanding of generally unsolvable problems by finding a tractable kernel formulation. The notes are mainly focused on practical applications such as data distillation and adversarial robustness, examples of inductive bias are also discussed.

Applying statistical learning theory to deep learning

Abstract Although statistical learning theory provides a robust framework to understand supervised learning, many theoretical aspects of deep learning remain unclear;in particular, how different architectures may lead to inductive bias when trained using gradient-based methods. The goal of these lectures is to provide an overview of some of the main questions that arise when attempting to understand deep learning from a learning theory perspective. After a brief reminder on statistical learning theory and stochastic optimization, we discuss implicit bias in the context of benign overfitting. We then move to a general description of the mirror descent algorithm, showing how we may go back and forth between a parameter space and the corresponding function space for a given learning problem, as well as how the geometry of the learning problem may be represented by a metric tensor. Building on this framework, we provide a detailed study of the implicit bias of gradient descent on linear diagonal networks for various regression tasks, showing how the loss function, scale of parameters at initialization and depth of the network may lead to various forms of implicit bias; in particular, transitioning between kernel and feature learning regimes.

Spatiotemporal implicit neural representation as a generalized traffic data learner

Spatiotemporal Traffic Data (STTD) measures the complex dynamical behaviors of the multiscale transportation system. Existing methods aim to reconstruct STTD using low-dimensional models. However, they are limited to data-specific dimensions or source-dependent patterns, restricting them from unifying representations. Here, we present a novel paradigm to address the STTD learning problem by parameterizing STTD as an implicit neural representation. To discern the underlying dynamics in low-dimensional regimes, coordinate-based neural networks that can encode high-frequency structures are employed to directly map coordinates to traffic variables. To unravel the entangled spatial–temporal interactions, the variability is decomposed into separate processes. We further enable modeling in irregular spaces such as sensor graphs using spectral embedding. Through continuous representations, our approach enables the modeling of a variety of STTD with a unified input, thereby serving as a generalized learner of the underlying traffic dynamics. It is also shown that it can learn implicit low-rank priors and smoothness regularization from the data, making it versatile for learning different dominating data patterns. We validate its effectiveness through extensive experiments in real-world scenarios, showcasing applications from corridor to network scales. Empirical results not only indicate that our model has significant superiority over conventional low-rank models, but also highlight that the versatility of the approach extends to different data domains, output resolutions, and network topologies. Comprehensive model analyses provide further insight into the inductive bias of STTD. We anticipate that this pioneering modeling perspective could lay the foundation for universal representation of STTD in various real-world tasks. PyTorch implementations of this project is publicly available at:https://github.com/tongnie/traffic_dynamics.

Swincloud: a hybrid network for cloud detection in thermal infrared remote sensing images

ABSTRACT Cloud cover is a significant factor affecting the effectiveness of satellite-based Earth observations. Existing cloud detection algorithms primarily rely on imaging data from satellite sensors in the visible to near-infrared spectral range, making it challenging to achieve day-and-night cloud monitoring. Convolutional neural networks have shown outstanding performance in previous cloud detection algorithms due to their robust ability to extract local information. However, their inherent inductive bias limits their capacity to learn long-range semantic information. To address these challenges, we proposed SwinCloud, a U-shaped semantic segmentation network based on an enhanced Swin Transformer for cloud detection in the thermal infrared spectral range. Specifically, we augment the Swin Transformer's window attention module with a CNN-based parallel pathway to effectively model global-local information. We employ a feature fusion module before the final upsampling module in the decoder to better integrate low-level spatial information and high-level semantic information. On the Landsat-8 cloud detection dataset, our model outperforms state-of-the-art methods. When transferred to the SDGSAT-TIS cloud detection dataset, the mIOU of experiment results reaches 69.9%, demonstrating the strong transferability of SwinCloud across different sensors. We also applied SwinCloud to cloud detection in the visible bands of Landsat-8. The results demonstrated SwinCloud's generalization capability across different bands.

Probing the effects of broken symmetries in machine learning

Abstract Symmetry is one of the most central concepts in physics, and it is no surprise that it has also been widely adopted as an inductive bias for machine-learning models applied to the physical sciences. 
This is especially true for models targeting the properties of matter at the atomic scale. Both established and state-of-the-art approaches, with almost no exceptions, are built to be exactly equivariant to translations, permutations, and rotations of the atoms.
Incorporating symmetries -- rotations in particular -- constrains the model design space and implies more complicated architectures that are often also computationally demanding. There are indications that unconstrained models can easily learn symmetries from data, and that doing so can even be beneficial for the accuracy of the model.
We demonstrate that an unconstrained architecture can be trained to achieve a high degree of rotational invariance, testing the impacts of the small symmetry breaking in realistic scenarios involving simulations of gas-phase, liquid, and solid water. 
We focus specifically on physical observables that are likely to be affected -- directly or indirectly -- by non-invariant behavior under rotations, finding negligible consequences when the model is used in an interpolative, bulk, regime. Even for extrapolative gas-phase predictions, the model remains very stable, even though symmetry artifacts are noticeable. 
We also discuss strategies that can be used to systematically reduce the magnitude of symmetry breaking when it occurs, and assess their impact on the convergence of observables.

HDKD: Hybrid data-efficient knowledge distillation network for medical image classification

Vision Transformers (ViTs) have achieved significant advancement in computer vision tasks due to their powerful modeling capacity. However, their performance notably degrades when trained with insufficient data due to lack of inherent inductive biases. Distilling knowledge and inductive biases from a Convolutional Neural Network (CNN) teacher has emerged as an effective strategy for enhancing the generalization of ViTs on limited datasets. Previous approaches to Knowledge Distillation (KD) have pursued two primary paths: some focused solely on distilling the logit distribution from CNN teacher to ViT student, neglecting the rich semantic information present in intermediate features due to the structural differences between them. Others integrated feature distillation along with logit distillation, yet this introduced alignment operations that limits the amount of knowledge transferred due to mismatched architectures and increased the computational overhead. To this end, this paper presents Hybrid Data-efficient Knowledge Distillation (HDKD) paradigm which employs a CNN teacher and a hybrid student. The choice of hybrid student serves two main aspects. First, it leverages the strengths of both convolutions and transformers while sharing the convolutional structure with the teacher model. Second, this shared structure enables the direct application of feature distillation without any information loss or additional computational overhead. Additionally, we propose an efficient light-weight convolutional block named Mobile Channel-Spatial Attention (MBCSA), which serves as the primary convolutional block in both teacher and student models. Extensive experiments on two medical public datasets showcase the superiority of HDKD over other state-of-the-art models and its computational efficiency. Source code at: https://github.com/omarsherif200/HDKD.

An inductive bias from quantum mechanics: learning order effects with non-commuting measurements

There are two major approaches to building good machine learning algorithms: feeding lots of data into large models or picking a model class with an “inductive bias” that suits the structure of the data. When taking the second approach as a starting point to design quantum algorithms for machine learning, it is important to understand how mathematical structures in quantum mechanics can lead to useful inductive biases in quantum models. In this work, we bring a collection of theoretical evidence from the quantum cognition literature to the field of quantum machine learning to investigate how non-commutativity of quantum observables can help to learn data with “order effects,” such as the changes in human answering patterns when swapping the order of questions in a survey. We design a multi-task learning setting in which a generative quantum model consisting of sequential learnable measurements can be adapted to a given task — or question order — by changing the order of observables, and we provide artificial datasets inspired by human psychology to carry out our investigation. Our first experimental simulations show that in some cases the quantum model learns more non-commutativity as the amount of order effect present in the data is increased and that the quantum model can learn to generate better samples for unseen question orders when trained on others — both signs that the model architecture suits the task.