Learning Bias Research Articles

Affine medical image registration with fusion feature mapping in local and global.

Objective. Medical image affine registration is a crucial basis before using deformable registration. On the one hand, the traditional affine registration methods based on step-by-step optimization are very time-consuming, so these methods are not compatible with most real-time medical applications. On the other hand, convolutional neural networks are limited in modeling long-range spatial relationships of the features due to inductive biases, such as weight sharing and locality. This is not conducive to affine registration tasks. Therefore, the evolution of real-time and high-accuracy affine medical image registration algorithms is necessary for registration applications.Approach. In this paper, we propose a deep learning-based coarse-to-fine global and local feature fusion architecture for fast affine registration, and we use an unsupervised approach for end-to-end training. We use multiscale convolutional kernels as our elemental convolutional blocks to enhance feature extraction. Then, to learn the long-range spatial relationships of the features, we propose a new affine registration framework with weighted global positional attention that fuses global feature mapping and local feature mapping. Moreover, the fusion regressor is designed to generate the affine parameters.Main results. The additive fusion method can be adaptive to global mapping and local mapping, which improves affine registration accuracy without the center of mass initialization. In addition, the max pooling layer and the multiscale convolutional kernel coding module increase the ability of the model in affine registration.Significance. We validate the effectiveness of our method on the OASIS dataset with 414 3D MRI brain maps. Comprehensive results demonstrate that our method achieves state-of-the-art affine registration accuracy and very efficient runtimes.

Training Object Detectors from Scratch: An Empirical Study in the Era of Vision Transformer

Modeling in computer vision has long been dominated by convolutional neural networks (CNNs). Recently, in light of the excellent performance of self-attention mechanism in the language field, transformers tailored for visual data have drawn significant attention and triumphed over CNNs in various vision tasks. These vision transformers heavily rely on large-scale pre-training to achieve competitive accuracy, which not only hinders the freedom of architectural design in downstream tasks like object detection, but also causes learning bias and domain mismatch in the fine-tuning stages. To this end, we aim to get rid of the “pre-train and fine-tune” paradigm of vision transformer and train transformer based object detector from scratch. Some earlier works in the CNNs era have successfully trained CNNs based detectors without pre-training, unfortunately, their findings do not generalize well when the backbone is switched from CNNs to a vision transformer. Instead of proposing a specific vision transformer based detector, in this work, our goal is to reveal the insights of training vision transformer based detectors from scratch. In particular, we expect those insights to help other researchers and practitioners, and inspire more interesting research in other fields, such as remote sensing, visual-linguistic pre-training, etc. One of the key findings is that both architectural changes and more epochs play critical roles in training vision transformer based detectors from scratch. Experiments on the MS COCO dataset demonstrate that vision transformer based detectors trained from scratch can also achieve similar performance to their counterparts with ImageNet pre-training.

Rational intuition: a study on semiotic subjectivity

Abstract Although scientific research usually emphasizes objectivity unaffected by any form of subjectivity, semiotic inquiries have to consider the role of subjectivity. In fact, social development, art, and literature always follow certain rules of objectivity, but these rules seem to be inescapably linked to the subjectivity of human beings. How can we avoid the subjectivity of research, or research biases caused by subjectivity, to ensure the objectivity of scientific research to the greatest possible extent? This paper answers the question through the cultural semiotics of jingshen, which is grounded in Chinese cultural traditions, especially the philosophies of Lao Tseu and Chuang Tzu. This line of inquiry seeks to identify objectivity in the constant transformation between the subject’s states of self and selflessness, while striving to avoid inductive and deductive biases caused by the “conceit” of rationality and to eliminate the constraints of specific concepts and images formed by existing knowledge as well as possible consequential “Verborgenheit.” The purpose is to achieve a state of “rational intuition,” returning to the natural mode of human existence and forging for semiotics a rationality-based path to the intuitive understanding of meaning.

Uncovering drone intentions using control physics informed machine learning

Unmanned Autonomous Vehicle (UAV) or drones are increasingly used across diverse application areas. Uncooperative drones do not announce their identity/flight plans and can pose a potential risk to critical infrastructures. Understanding drone’s intention is important to assigning risk and executing countermeasures. Intentions are often intangible and unobservable, and a variety of tangible intention classes are often inferred as a proxy. However, inference of drone intention classes using observational data alone is inherently unreliable due to observational and learning bias. Here, we developed a control-physics informed machine learning (CPhy-ML) that can robustly infer across intention classes. The CPhy-ML couples the representation power of deep learning with the conservation laws of aerospace models to reduce bias and instability. The CPhy-ML achieves a 48.28% performance improvement over traditional trajectory prediction methods. The reward inference results outperforms conventional inverse reinforcement learning approaches, decreasing the root mean squared spectral norm error from 3.3747 to 0.3229.

PCTC-Net: A Crack Segmentation Network with Parallel Dual Encoder Network Fusing Pre-Conv-Based Transformers and Convolutional Neural Networks.

Cracks are common defects that occur on the surfaces of objects and structures. Crack detection is a critical maintenance task that traditionally requires manual labor. Large-scale manual inspections are expensive. Research has been conducted to replace expensive human labor with cheaper computing resources. Recently, crack segmentation based on convolutional neural networks (CNNs) and transformers has been actively investigated for local and global information. However, the transformer is data-intensive owing to its weak inductive bias. Existing labeled datasets for crack segmentation are relatively small. Additionally, a limited amount of fine-grained crack data is available. To address this data-intensive problem, we propose a parallel dual encoder network fusing Pre-Conv-based Transformers and convolutional neural networks (PCTC-Net). The Pre-Conv module automatically optimizes each color channel with a small spatial kernel before the input of the transformer. The proposed model, PCTC-Net, was tested with the DeepCrack, Crack500, and Crackseg9k datasets. The experimental results showed that our model achieved higher generalization performance, stability, and F1 scores than the SOTA model DTrC-Net.

Evaluating Generalization, Bias, and Fairness in Deep Learning for Metal Surface Defect Detection: A Comparative Study

In recent years, deep learning models have led to improved accuracy in industrial defect detection, often using variants of YOLO (You Only Look Once), due to its high performance at a low cost. However, the generalizability, fairness and bias of their outcomes have not been examined, which may lead to overconfident predictions. Additionally, the complexity added by co-occurring defects, single and multi-class defects, and the effect on training, is not taken into consideration. This study addresses these critical gaps by introducing new methodologies for analyzing dataset complexity and evaluating model fairness. It introduces the novel approach of co-occurrence impact analysis, examining how the co-occurrence of defects in sample images affects performance, and introducing new dimensions to dataset preparation and training. Its aim is to increase model robustness in the face of real-world scenarios where multiple defects often appear together. Our study also innovates in the evaluation of model fairness by adapting the disparate impact ratio (DIR) to consider the true positive rate (TPR) across different groups and modifying the predictive parity difference (PPD) metric to focus on biases present in industrial quality control. Experiments demonstrate by cross-validation that the model trained on combined datasets significantly outperforms others in accuracy without overfitting and results in increased fairness, as validated by our novel fairness metrics. Explainability also provides valuable insights on the effects of different training regimes, notably absent in prior works. This work not only advances the field of deep learning for defect detection but also provides a strategic framework for future advancements, emphasizing the need for balanced datasets and considerations of ethics, fairness, bias and generalizability in the deployment of artificial intelligence in industry.

A Novel Method for Monocular Depth Estimation Using an Hourglass Neck Module.

In this paper, we propose a novel method for monocular depth estimation using the hourglass neck module. The proposed method has the following originality. First, feature maps are extracted from Swin Transformer V2 using a masked image modeling (MIM) pretrained model. Since Swin Transformer V2 has a different patch size for each attention stage, it is easier to extract local and global features from images input by the vision transformer (ViT)-based encoder. Second, to maintain the polymorphism and local inductive bias of the feature map extracted from Swin Transformer V2, a feature map is input into the hourglass neck module. Third, deformable attention can be used at the waist of the hourglass neck module to reduce the computation cost and highlight the locality of the feature map. Finally, the feature map traverses the neck and proceeds through a decoder, comprised of a deconvolution layer and an upsampling layer, to generate a depth image. To evaluate the objective reliability of the proposed method in this paper, we used the NYU Depth V2 dataset to compare and evaluate the methods published in other papers. As a result of the experiment, the RMSE value of the novel method for monocular depth estimation using the hourglass neck module proposed in this paper was 0.274, which was lower than those published in other papers. The lower the RMSE value, the better the depth estimation method; therefore, its efficiency compared to other techniques has been proven.

Physics-informed deep learning for multi-species membrane separations

Conventional continuum models for ion transport across polyamide membranes require solving partial differential equations (PDEs). These models typically introduce a host of assumptions and simplifications to improve the computational tractability of existing solvers. As a consequence of these constraints, conventional models struggle to generalize predictive performance to new unseen conditions. Deep learning has recently shown promise in alleviating many of these concerns, making it a promising avenue for surrogate models that can replace conventional PDE-based approaches. In this work, we develop a physics-informed deep learning model to predict ion transport across diverse membrane types. The proposed architecture leverages neural differential equations in conjunction with classical closure models as inductive biases directly encoded into the neural framework. The neural methods are pre-trained on simulated data from continuum models and fine-tuned on independent experiments to learn multi-ionic rejection behaviour. We also harness the attention mechanism, commonly observed in language modelling, to learn and infer key paired transport relationships. Gaussian noise augmentations from experimental uncertainty estimates are also introduced into the measured data to improve robustness and generalization. We study the neural framework’s performance relative to conventional PDE-based methods, and also compare the use of hard/soft inductive bias constraints on prediction accuracy. Lastly, we compare our approach to other competitive deep learning architectures and illustrate strong agreement with experimental measurements across all studied datasets.

AMSC-Net: Anatomy and multi-label semantic consistency network for semi-supervised fluid segmentation in retinal OCT

Automated segmentation of pathological fluid regions is crucial for digital diagnosis and individualized therapy under optical coherence tomography (OCT) images. Nonetheless, methods that rely on enormous annotations impede their clinical applications, as pixel-wise fine-grained labels are extraordinarily costly and require the expertise of ophthalmology. Though there exist works based on consistency-based and pseudo-label-based methods for reducing annotation reliance, the underutilization of consistency mechanisms and ignorance of fluid anatomical structures result in sub-optimal segmentation performance. In this work, we proposed a novel AMSC-Net devoted to semi-supervised fluid segmentation and achieves a 73.95% Dice score with 5% labeled data. Concretely, we developed a Heterogeneous Architecture Consistency (HAC) strategy based on our dual decoders enjoying different inductive biases. Moreover, we invented a Multi-label Semantic Consistency Loss (MSC-Loss) module in hierarchical semantic features and an Anatomy Contour Consistency Loss (ACC-Loss) module integrating anatomical constraint. These modules complementarily reinforce the quality of pseudo labels and boost semi-supervised training for robust segmentation results. To verify the superiority and bedside potential of our proposed AMSC-Net, we collected a large-scale fluid segmentation dataset composed of 22517 OCT images. Extensive quantitative and qualitative experiments validated the efficacy of our AMSC-Net with multiple novel techniques. Also, experimental results in a public fluid segmentation dataset demonstrated that our method achieves state-of-the-art performance. Code will be available at: https://github.com/ZeroOneGame/S4_Fluid_OCT.

Fading memory as inductive bias in residual recurrent networks

Residual connections have been proposed as an architecture-based inductive bias to mitigate the problem of exploding and vanishing gradients and increased task performance in both feed-forward and recurrent networks (RNNs) when trained with the backpropagation algorithm. Yet, little is known about how residual connections in RNNs influence their dynamics and fading memory properties. Here, we introduce weakly coupled residual recurrent networks (WCRNNs) in which residual connections result in well-defined Lyapunov exponents and allow for studying properties of fading memory. We investigate how the residual connections of WCRNNs influence their performance, network dynamics, and memory properties on a set of benchmark tasks. We show that several distinct forms of residual connections yield effective inductive biases that result in increased network expressivity. In particular, those are residual connections that (i) result in network dynamics at the proximity of the edge of chaos, (ii) allow networks to capitalize on characteristic spectral properties of the data, and (iii) result in heterogeneous memory properties. In addition, we demonstrate how our results can be extended to non-linear residuals and introduce a weakly coupled residual initialization scheme that can be used for Elman RNNs.

A causal perspective on dataset bias in machine learning for medical imaging

A causal perspective on dataset bias in machine learning for medical imaging

CDYL for infrared and visible light image dense small object detection

To address the phenomenon of many small and hard-to-detect objects in infrared and visible light images, we propose an object detection algorithm CDYL (Convolution to Fully Connect-ed-Deformable Convolution You only Look once) based on the CFC-DC (Convolution to Fully Connected-Deformable Convolution) module. The core operator of CDYL is CFC-DC, making our model not only have a large effective receptive field in infrared and visible light images, but also have adaptive spatial aggregation conditioned by input and task information. As a result, the CDYL reduces the strict inductive bias of traditional CNNs and has long-range dependence for large kernel convolution as well as adaptive spatial aggregation, deeply mining of edge and correlation information in images to enhance sensitivity to small objects, thereby improving performance in dense small object detection tasks. In order to improve the ability of the CFC-DC module to perceive the detailed information of the image, we use the Mish activation function, which has a wider minima which improves the generalization. The effectiveness as well as the generalization of CDYL is evaluated on an infrared image dataset and an UAV image dataset, and it is compared with other state-of-the-art object detection algorithms. Compared to the baseline network YOLOv8l, our model achieved a 3.0% improvement in mAP0.5 in infrared image detection tasks and a 1.1% improvement in mAP0.5 in visible light image detection tasks. The experimental results show that the proposed algorithm achieves superior average precision values on both infrared and visible light images, while maintaining a light weight. Code is publicly available at https://github.com/yangzhu1/CDYL.

MOTC: Abdominal Multi-objective Segmentation Model with Parallel Fusion of Global and Local Information.

Convolutional Neural Networks have been widely applied in medical image segmentation. However, the existence of local inductive bias in convolutional operations restricts the modeling of long-term dependencies. The introduction of Transformer enables the modeling of long-term dependencies and partially eliminates the local inductive bias in convolutional operations, thereby improving the accuracy of tasks such as segmentation and classification. Researchers have proposed various hybrid structures combining Transformer and Convolutional Neural Networks. One strategy is to stack Transformer blocks and convolutional blocks to concentrate on eliminating the accumulated local bias of convolutional operations. Another strategy is to nest convolutional blocks and Transformer blocks to eliminate bias within each nested block. However, due to the granularity of bias elimination operations, these two strategies cannot fully exploit the potential of Transformer. In this paper, a parallel hybrid model is proposed for segmentation, which includes a Transformer branch and a Convolutional Neural Network branch in encoder. After parallel feature extraction, inter-layer information fusion and exchange of complementary information are performed between the two branches, simultaneously extracting local and global features while eliminating the local bias generated by convolutional operations within the current layer. A pure convolutional operation is used in decoder to obtain final segmentation results. To validate the impact of the granularity of bias elimination operations on the effectiveness of local bias elimination, the experiments in this paper were conducted on Flare21 dataset and Amos22 dataset. The average Dice coefficient reached 92.65% on Flare21 dataset, and 91.61% on Amos22 dataset, surpassing comparative methods. The experimental results demonstrate that smaller granularity of bias elimination operations leads to better performance.

Impact of conditional modelling for a universal autoregressive quantum state

We present a generalized framework to adapt universal quantum state approximators, enabling them to satisfy rigorous normalization and autoregressive properties. We also introduce filters as analogues to convolutional layers in neural networks to incorporate translationally symmetrized correlations in arbitrary quantum states. By applying this framework to the Gaussian process state, we enforce autoregressive and/or filter properties, analyzing the impact of the resulting inductive biases on variational flexibility, symmetries, and conserved quantities. In doing so we bring together different autoregressive states under a unified framework for machine learning-inspired ansätze. Our results provide insights into how the autoregressive construction influences the ability of a variational model to describe correlations in spin and fermionic lattice models, as well as ab initio electronic structure problems where the choice of representation affects accuracy. We conclude that, while enabling efficient and direct sampling, thus avoiding autocorrelation and loss of ergodicity issues in Metropolis sampling, the autoregressive construction materially constrains the expressivity of the model in many systems.

Goal-conditioned offline reinforcement learning through state space partitioning

AbstractOffline reinforcement learning (RL) aims to create policies for sequential decision-making using exclusively offline datasets. This presents a significant challenge, especially when attempting to accomplish multiple distinct goals or outcomes within a given scenario while receiving sparse rewards. Prior methods using advantage weighting for offline goal-conditioned learning improve policies monotonically. However, they still face challenges from distribution shift and multi-modality that arise due to conflicting ways to reach a goal. This issue is especially challenging in long-horizon tasks, where the presence of multiple, often conflicting, solutions makes it hard to identify a single optimal policy for transitioning from a state to a desired goal. To address these challenges, we introduce a complementary advantage-based weighting scheme that incorporates an additional source of inductive bias. Given a value-based partitioning of the state space, the contribution of actions expected to lead to target regions that are easier to reach, compared to the final goal, is further increased. Our proposed approach, Dual-Advantage Weighted Offline Goal-conditioned RL, outperforms several competing offline algorithms in widely used benchmarks. Furthermore, we provide a theoretical guarantee that the learned policy will not be inferior to the underlying behavior policy.

Spatial transfer of object-based statistical learning

A large number of recent studies have demonstrated that efficient attentional selection depends to a large extent on the ability to extract regularities present in the environment. Through statistical learning, attentional selection is facilitated by directing attention to locations in space that were relevant in the past while suppressing locations that previously were distracting. The current study shows that we are not only able to learn to prioritize locations in space but also locations within objects independent of space. Participants learned that within a specific object, particular locations within the object were more likely to contain relevant information than other locations. The current results show that this learned prioritization was bound to the object as the learned bias to prioritize a specific location within the object stayed in place even when the object moved to a completely different location in space. We conclude that in addition to spatial attention prioritization of locations in space, it is also possible to learn to prioritize relevant locations within specific objects. The current findings have implications for the inferred spatial priority map of attentional weights as this map cannot be strictly retinotopically organized.

Learnability Advantage of Segmental Repetitions in Word Learning.

To date, research on wordform learning biases has mostly focused on language-dependent factors, such as the phonotactics and neighborhood density of the language(s) known by the learner. Domain-general biases, by contrast, have received little attention. In this study, we focus on one such bias-an advantage for string-internal repetitions-and examine its effects on wordform learning. Importantly, we consider whether any type of segmental repetition is equally beneficial for word recall, or whether learning is favored more or only by repeated consonants, in line with previous research indicating that consonants play a larger role than vowels in lexical processing. In Experiment 1, adult English speakers learned artificial consonant-vowel-consonant-vowel words containing either a repeated consonant (e.g., /sesu/, "c-rep"), a repeated vowel (e.g., /sepe/, "v-rep"), or dissimilar consonants and vowels (e.g., /sepu/, "no-rep"). Recall results showed no advantage for v-reps but higher accuracy for c-reps compared with no-reps. In Experiment 2, participants performed a label preference task with the same stimuli. The results showed dispreference for both c-reps and v-reps relative to no-reps, indicating that the results of Experiment 1 are independent of wordlikeness effects. These outcomes reveal that there is a form-learning bias for words with identical consonants but not for words with identical vowels, suggesting that a domain-general advantage for repetitions within strings is modulated by a language-specific processing bias for consonants.

Grounded language acquisition through the eyes and ears of a single child.

Starting around 6 to 9 months of age, children begin acquiring their first words, linking spoken words to their visual counterparts. How much of this knowledge is learnable from sensory input with relatively generic learning mechanisms, and how much requires stronger inductive biases? Using longitudinal head-mounted camera recordings from one child aged 6 to 25 months, we trained a relatively generic neural network on 61 hours of correlated visual-linguistic data streams, learning feature-based representations and cross-modal associations. Our model acquires many word-referent mappings present in the child's everyday experience, enables zero-shot generalization to new visual referents, and aligns its visual and linguistic conceptual systems. These results show how critical aspects of grounded word meaning are learnable through joint representation and associative learning from one child's input.

Knowledge-Enhanced Distributed Graph Autoencoder for Multiunit Industrial Plant-Wide Process Monitoring

In multiunit industrial plant-wide processes (MIPPs), data-driven distributed process monitoring methods have played an important role in ensuring process safety and reliability. However, the existing studies fail to leverage the relational knowledge about the underlying process structure of MIPPs, which may lead to inaccurate process modeling and degradation of the monitoring performance. To tackle these issues, this work proposes a novel knowledge-enhanced distributed graph autoencoder (KDGAE) for plant-wide process monitoring. Firstly, a posterior graph structure learning module (PGSL) is designed for relational knowledge discovery, which explicitly characterizes the dependencies between operation units in MIPPs into a directed graph. Prior knowledge is introduced into PGSL as a learning bias that steers the posterior graph to adhere to the underlying physics. Then, a novel distributed graph autoencoder (DGAE) is developed to encode both the local information within each unit and the global information between units for distributed process monitoring. The discovered posterior knowledge is embedded as a relational inductive bias in DGAE to enhance the capability of unsupervised representation learning, thereby improving the monitoring performance. Finally, the effectiveness of the proposed method is demonstrated through the Tennessee Eastman process and a real-world air separation process.

Brain tumor grade classification using the ConvNext architecture.

Brain tumor grade is an important aspect of brain tumor diagnosis and helps to plan for treatment. Traditional methods of diagnosis, including biopsy and manual examination of medical images, are either invasive or may result in inaccurate diagnoses. This study proposes a brain tumor grade classification technique using a modern convolutional neural network (CNN) architecture called ConvNext that inputs magnetic resonance imaging (MRI) data. Deep learning-based techniques are replacing invasive procedures for consistent, accurate, and non-invasive diagnosis of brain tumors. A well-known challenge of using deep learning architectures in medical imaging is data scarcity. Modern-day architectures have huge trainable parameters and require massive datasets to achieve the desired accuracy and avoid overfitting. Therefore, transfer learning is popular among researchers using medical imaging data. Recently, transformer-based architectures have surpassed CNNs for image data. However, recently proposed CNNs have achieved superior accuracy by introducing some tweaks inspired by vision transformers. This study proposed a technique to extract features from the ConvNext architecture and feed these features to a fully connected neural network for final classification. The proposed study achieved state-of-the-art performance on the BraTS 2019 dataset using pre-trained ConvNext. The best accuracy of 99.5% was achieved when three MRI sequences were input as three channels of the pre-trained CNN. The study demonstrated the efficacy of the representations learned by a modern CNN architecture, which has a higher inductive bias for the image data than vision transformers for brain tumor grade classification.