Learning Bias Research Articles

Patient teacher can impart locality to improve lightweight vision transformer on small dataset

Vision Transformer (ViT) has achieved unprecedented success in vision tasks with the assistance of abundant data. However, the lack of inductive bias in lightweight ViT makes learning locality challenging on small datasets, leading to poor performance. This limitation impedes the application of lightweight ViT in scenarios with limited datasets and computational power. Knowledge Distillation (KD) allows student models to benefit from the teacher model. However, in the progressive learning stage, traditional single-stage KD methods are usually suboptimal for delivering fixed knowledge to the growing student model. To address these issues, we propose a simple yet effective two-stage KD method called Curriculum Information Knowledge Distillation (CIKD) for the first time. Specifically, we incorporate a curriculum learning framework, progressing from easy to difficult, in the KD curriculum. At the first stage, i.e., Attention Locality Imitation (ALI), the student model learns locality from the low-level semantic features of the teacher model through self-attention distillation. Afterward, at the second stage, i.e., Logit Mimicking (LM), the student model learns label information and high-level semantic logit from the teacher model. Without bells and whistles, our approach achieves state-of-the-art results on 8 small-scale datasets with ViT-Tiny (5.0M). Our code and model weights are available at: https://github.com/newLLing/CIKD.

Capturing the Heterogeneity of Word Learners by Analyzing Persons.

Accurately capturing children's word learning abilities is critical for advancing our understanding of language development. Researchers have demonstrated that utilizing more complex statistical methods, such as mixed-effects regression and hierarchical linear modeling, can lead to a more complete understanding of the variability observed within children's word learning abilities. In the current paper, we demonstrate how a person-centered approach to data analysis can provide additional insights into the heterogeneity of word learning ability among children while also aiding researchers' efforts to draw individual-level conclusions. Using previously published data with 32 typically developing and 32 late-talking infants who completed a novel noun generalization (NNG) task to assess word learning biases (i.e., shape and material biases), we compare this person-centered method to three traditional statistical approaches: (1) a t-test against chance, (2) an analysis of variance (ANOVA), and (3) a mixed-effects regression. With each comparison, we present a novel question raised by the person-centered approach and show how results from the corresponding analyses can lead to greater nuance in our understanding of children's word learning capabilities. Person-centered methods, then, are shown to be valuable tools that should be added to the growing body of sophisticated statistical procedures used by modern researchers.

A 3D boundary-guided hybrid network with convolutions and Transformers for lung tumor segmentation in CT images

—Accurate lung tumor segmentation from Computed Tomography (CT) scans is crucial for lung cancer diagnosis. Since the 2D methods lack the volumetric information of lung CT images, 3D convolution-based and Transformer-based methods have recently been applied in lung tumor segmentation tasks using CT imaging. However, most existing 3D methods cannot effectively collaborate the local patterns learned by convolutions with the global dependencies captured by Transformers, and widely ignore the important boundary information of lung tumors. To tackle these problems, we propose a 3D boundary-guided hybrid network using convolutions and Transformers for lung tumor segmentation, named BGHNet. In BGHNet, we first propose the Hybrid Local-Global Context Aggregation (HLGCA) module with parallel convolution and Transformer branches in the encoding phase. To aggregate local and global contexts in each branch of the HLGCA module, we not only design the Volumetric Cross-Stripe Window Transformer (VCSwin-Transformer) to build the Transformer branch with local inductive biases and large receptive fields, but also design the Volumetric Pyramid Convolution with transformer-based extensions (VPConvNeXt) to build the convolution branch with multi-scale global information. Then, we present a Boundary-Guided Feature Refinement (BGFR) module in the decoding phase, which explicitly leverages the boundary information to refine multi-stage decoding features for better performance. Extensive experiments were conducted on two lung tumor segmentation datasets, including a private dataset (HUST-Lung) and a public benchmark dataset (MSD-Lung). Results show that BGHNet outperforms other state-of-the-art 2D or 3D methods in our experiments, and it exhibits superior generalization performance in both non-contrast and contrast-enhanced CT scans.

CSWin-UNet: Transformer UNet with cross-shaped windows for medical image segmentation

Deep learning, especially convolutional neural networks (CNNs) and Transformer architectures, have become the focus of extensive research in medical image segmentation, achieving impressive results. However, CNNs come with inductive biases that limit their effectiveness in more complex, varied segmentation scenarios. Conversely, while Transformer-based methods excel at capturing global and long-range semantic details, they suffer from high computational demands. In this study, we propose CSWin-UNet, a novel U-shaped segmentation method that incorporates the CSWin self-attention mechanism into the UNet to facilitate horizontal and vertical stripes self-attention. This method significantly enhances both computational efficiency and receptive field interactions. Additionally, our innovative decoder utilizes a content-aware reassembly operator that strategically reassembles features, guided by predicted kernels, for precise image resolution restoration. Our extensive empirical evaluations on diverse datasets, including synapse multi-organ CT, cardiac MRI, and skin lesions demonstrate that CSWin-UNet maintains low model complexity while delivering high segmentation accuracy.

Disentangled variational auto-encoder for multimodal fusion performance analysis in multimodal sentiment analysis

Multimodal Sentiment Analysis (MSA) holds extensive applicability owing to its capacity to analyze and interpret users' emotions, feelings, and perspectives by integrating complementary information from multiple modalities. However, inefficient and unbalanced cross-modal information fusion substantially undermines the accuracy and reliability of MSA models. Consequently, a critical challenge in the field now lies in effectively assessing the information integration capabilities of these models to ensure balanced and equitable processing of multimodal data. In this paper, a Disentanglement-based Variable Auto-Encoder (DVAE) is proposed for systematically assessing fusion performance and investigating the factors that facilitate multimodal fusion. Specifically, a distribution constraint module is presented to decouple the fusion matrices and generate multiple low-dimensional and trustworthy disentangled latent vectors that adhere to the authentic unimodal input distribution. In addition, a combined loss term is modified to effectively balance inductive bias, signal reconstruction, and distribution constraint items to facilitate the optimization of neural network weights and parameters. Utilizing the proposed evaluation method, we can evaluate the fusion performance of multimodal models by contrasting the classification degradation ratio derived from disentangled hidden representations and joint representations. Experiments conducted with eight state-of-the-art multimodal fusion methods on the CMU-MOSEI and CMU-MOSEI benchmark datasets demonstrate that DVAE is capable of effectively evaluating the effects of multimodal fusion. Moreover, the comparative experimental results indicate that the equalizing effect among various advanced mechanisms in multimodal sentiment analysis, as well as the single-peak characteristic of the ground label distribution, both contribute significantly to multimodal data fusion.

Seeing the Forest but Naming the Trees: An Object-Over-Place Bias in Learning Noun Labels.

Objects and places are foundational spatial domains represented in human symbolic expressions, like drawings, which show a prioritization of depicting small-scale object-shape information over the large-scale navigable place information in which objects are situated. Is there a similar object-over-place bias in language? Across six experiments, adults and 3- to 4-year-old children were asked either to extend a novel noun in a labeling phrase, to extend a novel noun in a prepositional phrase, or to simply match pictures. To dissociate specific object and place information from more general figure and ground information, participants either saw scenes with both place information (a room) and object information (a block in the room), or scenes with two kinds of object information that matched the figure-ground relations of the room and block by presenting an open container with a smaller block inside. While adults showed a specific object-over-place bias in both extending novel noun labels and matching, they did not show this bias in extending novel nouns following prepositions. Young children showed this bias in extending novel noun labels only. Spatial domains may thus confer specific and foundational biases for word learning that may change through development in a way that is similar to that of other word-learning biases about objects, like the shape bias. These results expand the symbolic scope of prior studies on object biases in drawing to object biases in language, and they expand the spatial domains of prior studies characterizing the language of objects and places.

Addressing bias in bagging and boosting regression models.

As artificial intelligence (AI) becomes widespread, there is increasing attention on investigating bias in machine learning (ML) models. Previous research concentrated on classification problems, with little emphasis on regression models. This paper presents an easy-to-apply and effective methodology for mitigating bias in bagging and boosting regression models, that is also applicable to any model trained through minimizing a differentiable loss function. Our methodology measures bias rigorously and extends the ML model's loss function with a regularization term to penalize high correlations between model errors and protected attributes. We applied our approach to three popular tree-based ensemble models: a random forest model (RF), a gradient-boosted model (GBT), and an extreme gradient boosting model (XGBoost). We implemented our methodology on a case study for predicting road-level traffic volume, where RF, GBT, and XGBoost models were shown to have high accuracy. Despite high accuracy, the ML models were shown to perform poorly on roads in minority-populated areas. Our bias mitigation approach reduced minority-related bias by over 50%.

Business Cycle Variations in Manager and Investor Sentiment Indices

I find that the highly regarded investor and manager sentiment indices demonstrate both cyclical and persistent variations. The presence of cyclicality in the orthogonalized indices suggests the existence of feedback loops connecting sentiment with economic or market outcomes. For example, optimistic sentiment can stimulate increased consumer spending and investment, thereby fostering economic growth and further boosting sentiment. These feedback loops have the potential to amplify cyclical sentiment trends and increase the volatility of economic and market cycles, leading to persistent cycles driven by feedback mechanisms. Importantly, my results remain robust against potential mean reversion resulting from heuristic behaviors such as herding and overreaction, as well as against random behavior arising from intermittent bubbles characterized by near-rational learning and potential overextrapolation bias.

DiRecNetV2: A Transformer-Enhanced Network for Aerial Disaster Recognition

The integration of Unmanned Aerial Vehicles (UAVs) with artificial intelligence (AI) models for aerial imagery processing in disaster assessment, necessitates models that demonstrate exceptional accuracy, computational efficiency, and real-time processing capabilities. Traditionally Convolutional Neural Networks (CNNs), demonstrate efficiency in local feature extraction but are limited by their potential for global context interpretation. On the other hand, Vision Transformers (ViTs) show promise for improved global context interpretation through the use of attention mechanisms, although they still remain underinvestigated in UAV-based disaster response applications. Bridging this research gap, we introduce DiRecNetV2, an improved hybrid model that utilizes convolutional and transformer layers. It merges the inductive biases of CNNs for robust feature extraction with the global context understanding of Transformers, maintaining a low computational load ideal for UAV applications. Additionally, we introduce a new, compact multi-label dataset of disasters, to set an initial benchmark for future research, exploring how models trained on single-label data perform in a multi-label test set. The study assesses lightweight CNNs and ViTs on the AIDERSv2 dataset, based on the frames per second (FPS) for efficiency and the weighted F1 scores for classification performance. DiRecNetV2 not only achieves a weighted F1 score of 0.964 on a single-label test set but also demonstrates adaptability, with a score of 0.614 on a complex multi-label test set, while functioning at 176.13 FPS on the Nvidia Orin Jetson device.

Multimodal Religiously Hateful Social Media Memes Classification Based on Textual and Image Data

Multimodal hateful social media meme detection is an important and challenging problem in the vision-language domain. Recent studies show high accuracy for such multimodal tasks due to datasets that provide better joint multimodal embedding to narrow the semantic gap. Religiously hateful meme detection is not extensively explored among published datasets. While there is a need for higher accuracy on religiously hateful memes, deep learning–based models often suffer from inductive bias. This issue is addressed in this work with the following contributions. First, a religiously hateful memes dataset is created and published publicly to advance hateful religious memes detection research. Over 2000 meme images are collected with their corresponding text. The proposed approach compares and fine-tunes VisualBERT pre-trained on the Conceptual Caption (CC) dataset for the downstream classification task. We also extend the dataset with the Facebook hateful memes dataset. We extract visual features using ResNeXT-152 Aggregated Residual Transformations–based Masked Regions with Convolutional Neural Networks (R-CNN) and Bidirectional Encoder Representations from Transformers (BERT) uncased for textual encoding for the early fusion model. We use the primary evaluation metric of an Area Under the Operator Characters Curve (AUROC) to measure model separability. Results show that the proposed approach has a higher AUROC score of 78%, proving the model’s higher separability performance and an accuracy of 70%. It shows comparatively superior performance considering dataset size and against ensemble-based machine learning approaches.

Lightweight Single Image Super-Resolution via Efficient Mixture of Transformers and Convolutional Networks.

In this paper, we propose a Local Global Union Network (LGUN), which effectively combines the strengths of Transformers and Convolutional Networks to develop a lightweight and high-performance network suitable for Single Image Super-Resolution (SISR). Specifically, we make use of the advantages of Transformers to provide input-adaptation weighting and global context interaction. We also make use of the advantages of Convolutional Networks to include spatial inductive biases and local connectivity. In the shallow layer, the local spatial information is encoded by Multi-order Local Hierarchical Attention (MLHA). In the deeper layer, we utilize Dynamic Global Sparse Attention (DGSA), which is based on the Multi-stage Token Selection (MTS) strategy to model global context dependencies. Moreover, we also conduct extensive experiments on both natural and satellite datasets, acquired through optical and satellite sensors, respectively, demonstrating that LGUN outperforms existing methods.

YOLO-HyperVision: A vision transformer backbone-based enhancement of YOLOv5 for detection of dynamic traffic information

With the increase of traffic flow in modern urban areas, traffic congestion has become a serious problem that affects people’s normal production and life. Using target detection technology instead of manual labor can quickly detect the road traffic situation and provide timely information about the traffic flow. However, when using drones to observe the traffic flow in the air, the perspective effect will cause the detected vehicles and pedestrians to be very small, and the scale difference between different categories of targets is large, which increases the detection difficulty of a single convolutional neural network model. In order to solve the problem of low accuracy of traditional single-stage target detection models, this study proposes an improved Yolov5 vehicle target detection model with Vision Transformer (VIT) backbone, You Only Look Once-HyperVision (YOLO-HV), which aims to solve the problem of poor multi-scale target recognition performance caused by the inability of traditional CNN networks to integrate contextual information, and help drones achieve more efficient and accurate traffic flow recognition functions. This study deeply integrates the Vision Transformer (VIT) backbone and Convolutional Neural Network (CNN), effectively combining the multi-scale detection advantages of Vision Transformer and the inductive bias ability of Convolutional Neural Network, and adds multi-scale residual modules and context correlation enhancement modules, which greatly improves the recognition accuracy of single-stage detectors for drone images. Through comparative experiments on the VisDrone dataset, it is found that the detection performance of this model is improved compared with several commonly used detection models. YOLO-HV can increase the mean average precision (mAP) by 3.3% compared with the pure convolutional network of the same model size. YOLO-HV model has achieved excellent performance in the task of traffic flow image detection taken by drones, and can more accurately identify and classify road vehicles than various target detection models.

Unsupervised deep learning bias correction of CMIP6 global ensemble precipitation predictions with cycle generative adversarial network

Abstract Climate change significantly impacts agricultural production, ecosystem stability, and socioeconomic development. Global climate models (GCMs) serve as the primary tool for simulating historical and future precipitation patterns. However, due to issues such as coarse resolution, boundary condition, and parameterization, model outputs require bias correction (BC). With the evolution of deep learning techniques, supervised convolutional neural network (CNN) frameworks have gained popularity in the area of climate model BC but face limitations in spatial correlation assumptions and data sparsity, particularly for extreme precipitation This study proposed an unsupervised learning approach using cycle generative adversarial network (CycleGAN) to correct the ensemble mean bias of models and compare its performance with CNN and Quantile Mapping methods. The results demonstrate that the proposed CycleGAN approach outperforms both CNN and Quantile Mapping in ensemble mean BC. It effectively learns the overall distribution of precipitation through an adversarial process and yields better extreme precipitation predictions.

The minimal computational substrate of fluid intelligence

The quantification of cognitive powers rests on identifying a behavioural task that depends on them. Such dependence cannot be assured, for the powers a task invokes cannot be experimentally controlled or constrained a priori, resulting in unknown vulnerability to failure of specificity and generalisability. Evaluating a compact version of Raven's Advanced Progressive Matrices (RAPM), a widely used clinical test of fluid intelligence, we show that LaMa, a self-supervised artificial neural network trained solely on the completion of partially masked images of natural environmental scenes, achieves representative human-level test scores a prima vista, without any task-specific inductive bias or training. Compared with cohorts of healthy and focally lesioned participants, LaMa exhibits human-like variation with item difficulty, and produces errors characteristic of right frontal lobe damage under degradation of its ability to integrate global spatial patterns. LaMa's narrow training and limited capacity suggest matrix-style tests may be open to computationally simple solutions that need not necessarily invoke the substrates of reasoning.

Wide-field imaging and recognition through cascaded complex scattering media

Considering the obvious application value in the field of minimally invasive and non-destructive clinical healthcare, we explore the challenge of wide-field imaging and recognition through cascaded complex scattering media, a topic that has been less researched, by realizing wide-field imaging and pathological screening through multimode fibers (MMF) and turbid media. To address the challenge of extracting features from chaotic and globally correlated speckles formed by transmitting images through cascaded complex scattering media, we establish a deep learning approach based on SMixerNet. By efficiently using the parameter-free matrix transposition, SMixerNet achieves a broad receptive field with less inductive bias through concise multi-layer perceptron (MLP). This approach circumvents the parameter's intensive requirements of previous implementations relying on self-attention mechanisms for global receptive fields. Imaging and pathological screening results based on extensive datasets demonstrate that our approach achieves better performance with fewer learning parameters, which helps deploy deep learning models on desktop-level edge computing devices for clinical healthcare. Our research shows that, deep learning facilitates imaging and recognition through cascaded complex scattering media. This research extends the scenarios of medical and industrial imaging, offering additional possibilities in minimally invasive and non-destructive clinical healthcare and industrial monitoring in harsh and complex scenarios.

Lightweight image super-resolution reconstruction based on mixed attention and global inductive bias network

Lightweight image super-resolution reconstruction based on mixed attention and global inductive bias network

A Machine Learning Bias Correction on Large‐Scale Environment of High‐Impact Weather Systems in E3SM Atmosphere Model

AbstractLarge‐scale dynamical and thermodynamical processes are common environmental drivers of high‐impact weather systems causing extreme weather events. However, such large‐scale environmental conditions often display systematic biases in climate simulations, posing challenges to evaluating high‐impact weather systems and extreme weather events. In this paper, a machine learning (ML) approach was employed to bias correct the large‐scale wind, temperature, and humidity simulated by the atmospheric component of the Energy Exascale Earth System Model (E3SM) at ∼1° resolution. The usefulness of the ML approach for extreme weather analysis was demonstrated with a focus on three high‐impact weather systems, including tropical cyclones (TCs), extratropical cyclones (ETCs), and atmospheric rivers (ARs). We show that the ML model can effectively reduce climate bias in large‐scale wind, temperature, and humidity while preserving their responses to imposed climate change perturbations. The bias correction is found to directly improve water vapor transport associated with ARs, and representations of thermodynamical flows associated with ETCs. When the bias‐corrected large‐scale winds are used to drive a synthetic TC track forecast model over the Atlantic basin, the resulting TC track density agrees better with that of the TC track model driven by observed winds. In addition, the ML model insignificantly interferes with the mean climate change signals of large‐scale storm environments as well as the occurrence and intensity of three weather systems. This study suggests that the proposed ML approach can be used to improve the downscaling of extreme weather events by providing more realistic large‐scale storm environments simulated by low‐resolution climate models.

Research on the performance of hybrid vision models based on ViT

ViT is a model proposed by the Google team in 2020 to apply a transformer in image classification, although it is not the first paper to apply a transformer in visual tasks, because of its model is simple and effective, and scalable, it has become a milestone work in the application of transformer in CV field, and has also triggered the subsequent related research. The core conclusion of the original ViT paper is that when there is enough data for pre-training, ViT outperforms CNN, breaks through the limitation of the transformer's lack of inductive bias, and can achieve better migration results in downstream tasks. However, when the training dataset is not large enough, ViT usually performs worse than ResNets of the same size, because Transformer lacks inductive bias, a kind of a priori knowledge, assumptions made in advance, compared with CNN. Through its innovative architecture and powerful performance, the visual representation transform (ViT) model continues to advance the field of computer vision, while facing some challenges and room for improvement. With the deepening of research and the continuous development of technology, ViT is expected to play a greater role in more practical applications. The article aims to explore the advantages and applicability of the ViT model and tries to construct a hybrid visual model to improve its generalization ability for different types of datasets, demonstrating the hybrid model's significance in improving the performance of the ViT model.

Neural network extrapolation to distant regions of the protein fitness landscape

Machine learning (ML) has transformed protein engineering by constructing models of the underlying sequence-function landscape to accelerate the discovery of new biomolecules. ML-guided protein design requires models, trained on local sequence-function information, to accurately predict distant fitness peaks. In this work, we evaluate neural networks’ capacity to extrapolate beyond their training data. We perform model-guided design using a panel of neural network architectures trained on protein G (GB1)-Immunoglobulin G (IgG) binding data and experimentally test thousands of GB1 designs to systematically evaluate the models’ extrapolation. We find each model architecture infers markedly different landscapes from the same data, which give rise to unique design preferences. We find simpler models excel in local extrapolation to design high fitness proteins, while more sophisticated convolutional models can venture deep into sequence space to design proteins that fold but are no longer functional. We also find that implementing a simple ensemble of convolutional neural networks enables robust design of high-performing variants in the local landscape. Our findings highlight how each architecture’s inductive biases prime them to learn different aspects of the protein fitness landscape and how a simple ensembling approach makes protein engineering more robust.

TransDiffSeg: Transformer-Based Conditional Diffusion Segmentation Model for Abdominal Multi-Objective.

In the domain of medical image segmentation, traditional diffusion probabilistic models are hindered by local inductive biases stemming from convolutional operations, constraining their ability to model long-term dependencies and leading to inaccurate mask generation. Conversely, Transformer offers a remedy by obviating the local inductive biases inherent in convolutional operations, thereby enhancing segmentation precision. Currently, the integration of Transformer and convolution operations mainly occurs in two forms: nesting and stacking. However, both methods address the bias elimination at a relatively large granularity, failing to fully leverage the advantages of both approaches. To address this, this paper proposes a conditional diffusion segmentation model named TransDiffSeg, which combines Transformer with convolution operations from traditional diffusion models in a parallel manner. This approach eliminates the accumulated local inductive bias of convolution operations at a finer granularity within each layer. Additionally, an adaptive feature fusion block is employed to merge conditional semantic features and noise features, enhancing global semantic information and reducing the Transformer's sensitivity to noise features. To validate the impact of granularity in bias elimination on performance and the impact of Transformer in alleviating the accumulated local inductive biases of convolutional operations in diffusion probabilistic models, experiments are conducted on the AMOS22 dataset and BTCV dataset. Experimental results demonstrate that eliminating local inductive bias at a finer granularity significantly improves the segmentation performance of diffusion probabilistic models. Furthermore, the results confirm that the finer the granularity of bias elimination, the better the segmentation performance.