Hard Examples Research Articles

Automated quantification of periodic discharges in human electroencephalogram

Periodic discharges (PDs) are pathologic patterns of epileptiform discharges repeating at regular intervals, commonly detected in the human electroencephalogram (EEG) signals in patients who are critically ill. The frequency and spatial extent of PDs are associated with the tendency of PDs to cause brain injury, existing automated algorithms do not quantify the frequency and spatial extent of PDs. The present study presents an algorithm for quantifying frequency and spatial extent of PDs. The algorithm quantifies the evolution of these parameters within a short (10–14 second) window, with a focus on lateralized and generalized periodic discharges. We test our algorithm on 300 ‘easy’, 300 ‘medium’, and 240 ‘hard’ examples (840 total epochs) of periodic discharges as quantified by interrater consensus from human experts when analyzing the given EEG epochs. We observe 95.0% agreement with a 95% confidence interval (CI) of [94.9%, 95.1%] between algorithm outputs with reviewer clincal judgement for easy examples, 92.0% agreement (95% CI [91.9%, 92.2%]) for medium examples, and 90.4% agreement (95% CI [90.3%, 90.6%]) for hard examples. The algorithm is also computationally efficient and is able to run in 0.385 ± 0.038 seconds for a single epoch using our provided implementation of the algorithm. The results demonstrate the algorithm’s effectiveness in quantifying these discharges and provide a standardized and efficient approach for PD quantification as compared to existing manual approaches.

A high-efficiency local and global detector for diatom-based drowning diagnosis

Detection and classification of diatoms has been considered as the most effective method for the drowning diagnosis in forensic practice. However, challenges (such as limited samples, complex background interference, and detecting multiple objects) remain during detection and classification. To address these challenges, we propose a MDMS (multi-scale dynamic multi-head self-attention)-based deep learning network for the diatom-based drowning diagnosis. This framework uses TFFT (two-stage full fine-tuning training strategy) to solve the challenges of limited diatom samples and detecting multiple objects, and builds a multi-scale dynamic multi-head self-attention module to extract the local and global features of diatom images. In the meantime, we introduce an online hard example mining strategy to attenuate the complex background interference. The experimental results show that the proposed framework can effectively reduce the missing and false detection rates of diatom objects, with the mAP (mean Average Precision) reaching 92.434%, which is better than the result using the mainstream methods.

A Multi-Scale Target Detection Method Using an Improved Faster Region Convolutional Neural Network Based on Enhanced Backbone and Optimized Mechanisms.

Currently, existing deep learning methods exhibit many limitations in multi-target detection, such as low accuracy and high rates of false detection and missed detections. This paper proposes an improved Faster R-CNN algorithm, aiming to enhance the algorithm's capability in detecting multi-scale targets. This algorithm has three improvements based on Faster R-CNN. Firstly, the new algorithm uses the ResNet101 network for feature extraction of the detection image, which achieves stronger feature extraction capabilities. Secondly, the new algorithm integrates Online Hard Example Mining (OHEM), Soft non-maximum suppression (Soft-NMS), and Distance Intersection Over Union (DIOU) modules, which improves the positive and negative sample imbalance and the problem of small targets being easily missed during model training. Finally, the Region Proposal Network (RPN) is simplified to achieve a faster detection speed and a lower miss rate. The multi-scale training (MST) strategy is also used to train the improved Faster R-CNN to achieve a balance between detection accuracy and efficiency. Compared to the other detection models, the improved Faster R-CNN demonstrates significant advantages in terms of mAP@0.5, F1-score, and Log average miss rate (LAMR). The model proposed in this paper provides valuable insights and inspiration for many fields, such as smart agriculture, medical diagnosis, and face recognition.

Non-relevant segment recognition via hard example mining under sparsely distributed events

We propose on/offline hard example mining (HEM) techniques to alleviate the degradation of the generalization performance in the sparse distribution of events in non-relevant segment (NRS) recognition and to examine their utility for long-duration surgery. Through on/offline HEM, higher recognition performance can be achieved by extracting hard examples that help train NRS events, for a given training dataset. Furthermore, we provide two performance measurement metrics to quantitatively evaluate NRS recognition in the clinical field. The existing precision and recall-based performance measurement method provides accurate quantitative statistics. However, it is not an efficient evaluation metric in tasks where false positive recognition errors are fatal, such as NRS recognition. We measured the false discovery rate (FDR) and threat score (TS) to provide quantitative values that meet the needs of the clinical setting. Finally, unlike previous studies, the utility of NRS recognition was improved by applying our model to long-duration surgeries, instead of short-length surgical operations such as cholecystectomy. In addition, the proposed training methodology was applied to robotic and laparoscopic surgery datasets to verify that it can be robustly applied to various clinical environments.

Contrast learning with hard example mining for few-shot fault diagnosis of rolling bearings

Contrast learning with hard example mining for few-shot fault diagnosis of rolling bearings

Two-Stage Adaptive Network for Semi-Supervised Cross-Domain Crater Detection under Varying Scenario Distributions

Crater detection can provide valuable information for humans to explore the topography and understand the history of extraterrestrial planets. Due to the significantly varying scenario distributions, existing detection models trained on known labelled crater datasets are hardly effective when applied to new unlabelled planets. To address this issue, we propose a two-stage adaptive network (TAN) for semi-supervised cross-domain crater detection. Our network is built on the YOLOv5 detector, where a series of strategies are employed to enhance its cross-domain generalisation ability. In the first stage, we propose an attention-based scale-adaptive fusion (ASAF) strategy to handle objects with significant scale variances. Furthermore, we propose a smoothing hard example mining (SHEM) loss function to address the issue of overfitting on hard examples. In the second stage, we propose a sort-based pseudo-labelling fine-tuning (SPF) strategy for semi-supervised learning to mitigate the distributional differences between source and target domains. For both stages, we employ weak or strong image augmentation to suit different cross-domain tasks. Experimental results on benchmark datasets demonstrate that the proposed network can enhance domain adaptation ability for crater detection under varying scenario distributions.

Penalizing the Hard Example But Not Too Much: A Strong Baseline for Fine-Grained Visual Classification.

Though significant progress has been achieved on fine-grained visual classification (FGVC), severe overfitting still hinders model generalization. A recent study shows that hard samples in the training set can be easily fit, but most existing FGVC methods fail to classify some hard examples in the test set. The reason is that the model overfits those hard examples in the training set, but does not learn to generalize to unseen examples in the test set. In this article, we propose a moderate hard example modulation (MHEM) strategy to properly modulate the hard examples. MHEM encourages the model to not overfit hard examples and offers better generalization and discrimination. First, we introduce three conditions and formulate a general form of a modulated loss function. Second, we instantiate the loss function and provide a strong baseline for FGVC, where the performance of a naive backbone can be boosted and be comparable with recent methods. Moreover, we demonstrate that our baseline can be readily incorporated into the existing methods and empower these methods to be more discriminative. Equipped with our strong baseline, we achieve consistent improvements on three typical FGVC datasets, i.e., CUB-200-2011, Stanford Cars, and FGVC-Aircraft. We hope the idea of moderate hard example modulation will inspire future research work toward more effective fine-grained visual recognition.

UCTNet: Uncertainty-guided CNN-Transformer hybrid networks for medical image segmentation

Transformer, born for long-range dependency establishment, has been widely studied as a complementary of convolutional neural networks (CNNs) in medical image segmentation. However, existing CNN-Transformer hybrid approaches simply pursue implicit feature fusion without considering their underlying functional overlap. Medical images typically follow stable anatomical structures, making convolution capable of handling most segmentation targets. Without differentiation, enforcing transformers to operate self-attention for all image patches would result in severe redundancy, hindering global feature extraction. In this paper, we propose a simple yet effective hybrid network named UCTNet where transformers only focus on establishing global dependency for CNN’s unreliable regions predicted through uncertainty estimation. In this way, CNN and transformer are explicitly fused to minimize functional overlap. More importantly, with fewer regions to handle, UCTNet is of better convergence to learn more robust feature representations for hard examples. Extensive experiments on publicly-available datasets demonstrate the superiority of UCTNet against the state-of-the-art approaches, achieving 89.44%, 92.91%, and 91.15% in Dice similarity coefficient on Synapse, ACDC, and ISIC2018 respectively. Furthermore, such a CNN-Transformer hybrid strategy is highly extendable to other frameworks without introducing additional computational burdens. Code is available at https://github.com/innocence0206/UCTNet.

PREFER: Prompt Ensemble Learning via Feedback-Reflect-Refine

As an effective tool for eliciting the power of Large Language Models (LLMs), prompting has recently demonstrated unprecedented abilities across a variety of complex tasks. To further improve the performance, prompt ensemble has attracted substantial interest for tackling the hallucination and instability of LLMs. However, existing methods usually adopt a two-stage paradigm, which requires a pre-prepared set of prompts with substantial manual effort, and is unable to perform directed optimization for different weak learners. In this paper, we propose a simple, universal, and automatic method named PREFER (Prompt Ensemble learning via Feedback-Reflect-Refine) to address the stated limitations. Specifically, given the fact that weak learners are supposed to focus on hard examples during boosting, PREFER builds a feedback mechanism for reflecting on the inadequacies of existing weak learners. Based on this, the LLM is required to automatically synthesize new prompts for iterative refinement. Moreover, to enhance stability of the prompt effect evaluation, we propose a novel prompt bagging method involving forward and backward thinking, which is superior to majority voting and is beneficial for both feedback and weight calculation in boosting. Extensive experiments demonstrate that our PREFER achieves state-of-the-art performance in multiple types of tasks by a significant margin. We have made our code publicly available.

HACDR-Net: Heterogeneous-Aware Convolutional Network for Diabetic Retinopathy Multi-Lesion Segmentation

Diabetic Retinopathy (DR), the leading cause of blindness in diabetic patients, is diagnosed by the condition of retinal multiple lesions. As a difficult task in medical image segmentation, DR multi-lesion segmentation faces the main concerns as follows. On the one hand, retinal lesions vary in location, shape, and size. On the other hand, because some lesions occupy only a very small part of the entire fundus image, the high proportion of background leads to difficulties in lesion segmentation. To solve the above problems, we propose a heterogeneous-aware convolutional network (HACDR-Net) that composes heterogeneous cross-convolution, heterogeneous modulated deformable convolution, and optional near-far-aware convolution. Our network introduces an adaptive aggregation module to summarize the heterogeneous feature maps and get diverse lesion areas in the heterogeneous receptive field along the channels and space. In addition, to solve the problem of the highly imbalanced proportion of focal areas, we design a new medical image segmentation loss function, Noise Adjusted Loss (NALoss). NALoss balances the predictive feature distribution of background and lesion by jointing Gaussian noise and hard example mining, thus enhancing awareness of lesions. We conduct the experiments on the public datasets IDRiD and DDR, and the experimental results show that the proposed method achieves better performance than other state-of-the-art methods. The code is open-sourced on github.com/xqh180110910537/HACDR-Net.

AdvMix: Adversarial Mixing Strategy for Unsupervised Domain Adaptive Object Detection

Recent object detection networks suffer from performance degradation when training data and test data are distinct in image styles and content distributions. In this paper, we propose a domain adaptive method, Adversarial Mixing (AdvMix), where the label-rich source domain and unlabeled target domain are jointly trained by the adversarial feature alignment and a self-training strategy. To diminish the style gap, we design the Adversarial Gradient Reversal Layer (AdvGRL), containing a global-level domain discriminator to align the domain features by gradient reversal, and an adversarial weight mapping function to enhance the stability of domain-invariant features by hard example mining. To eliminate the content gap, we introduce a region mixing self-supervised training strategy where a region of the target image with the highest confidence is selected to merge with the source image, and the synthesis image is self-supervised by the consistency loss. To improve the reliability of self-training, we propose a strict confidence metric combining both object and bounding box uncertainty. Extensive experiments conducted on three benchmarks demonstrate that AdvMix achieves prominent performance in terms of detection accuracy, surpassing existing domain adaptive methods by nearly 5% mAP.

Key point calibrating and clustering for hard example mining of dense analogs in the fish-eye lens.

Recent years have witnessed widespread applications of the fish-eye lens with a wide field-of-view. However, its inherent distortion poses a big challenge to the intelligent recognition of dense analogs (IRDA) by convolutional neural networks (CNN). The major bottleneck of existing CNN models lies in their limited modeling capacity for distorted objects in fish-eye images, leading to the misclassification of hard examples. To further improve the accuracy of IRDA, we propose a novel key point calibrating and clustering (KPCC) algorithm based on the hemispherical projection model. Our method can effectively correct the hard example misclassification predicted by the CNN, significantly enhancing the performance of the IRDA. The experiments show that, as a light-weight computation calibrating and stable adaptive clustering method, the KPCC increases the precision and recall rate of IRDA on the intelligent retail dataset by 8.55% and 8.07%, respectively; compared with the classic Focalloss, QFocalloss, and OHEM (online hard example mining), it can mine hard examples more sufficiently, especially in the scene of distorted dense analog detection.

One-Bit Supervision for Image Classification: Problem, Solution, and Beyond

This article presents one-bit supervision, a novel setting of learning with fewer labels, for image classification. Instead of the training model using the accurate label of each sample, our setting requires the model to interact with the system by predicting the class label of each sample and learn from the answer whether the guess is correct, which provides one bit (yes or no) of information. An intriguing property of the setting is that the burden of annotation largely is alleviated in comparison to offering the accurate label. There are two keys to one-bit supervision: (i) improving the guess accuracy and (ii) making good use of the incorrect guesses. To achieve these goals, we propose a multi-stage training paradigm and incorporate negative label suppression into an off-the-shelf semi-supervised learning algorithm. Theoretical analysis shows that one-bit annotation is more efficient than full-bit annotation in most cases and gives the conditions of combining our approach with active learning. Inspired by this, we further integrate the one-bit supervision framework into the self-supervised learning algorithm, which yields an even more efficient training schedule. Different from training from scratch, when self-supervised learning is used for initialization, both hard example mining and class balance are verified to be effective in boosting the learning performance. However, these two frameworks still need full-bit labels in the initial stage. To cast off this burden, we utilize unsupervised domain adaptation to train the initial model and conduct pure one-bit annotations on the target dataset. In multiple benchmarks, the learning efficiency of the proposed approach surpasses that using full-bit, semi-supervised supervision.

Efficient History-Driven Adversarial Perturbation Distribution Learning in Low Frequency Domain

The existence of adversarial image makes us have to doubt the credibility of artificial intelligence system. Attackers can use carefully processed adversarial images to carry out a variety of attacks. Inspired by the theory of image compressed sensing, this paper proposes a new black-box attack, \(\mathcal {N}\text{-HSA}_{LF}\) . It uses covariance matrix adaptive evolution strategy (CMA-ES) to learn the distribution of adversarial perturbation in low frequency domain, reducing the dimensionality of solution space. And sep-CMA-ES is used to set the covariance matrix as a diagonal matrix, which further reduces the dimensions that need to be updated for the covariance matrix of multivariate Gaussian distribution learned in attacks, thereby reducing the computational cost of attack. And on this basis, we propose history-driven mean update and current optimal solution-guided improvement strategies to avoid the evolution of distribution to a worse direction. The experimental results show that the proposed \(\mathcal {N}\text{-HSA}_{LF}\) can achieve a higher attack success rate with fewer queries on attacking both CNN-based and transformer-based target models under \(L_2\) -norm and \(L_\infty\) -norm constraints of perturbation. We also conduct an ablation study and the results show that the proposed improved strategies can effectively reduce the number of visits to the target model when making adversarial examples for hard examples. In addition, our attack is able to make the integrated defense strategy of GRIP-GAN and noise-embedded training ineffective to a certain extent.

InfoAT: Improving Adversarial Training Using the Information Bottleneck Principle.

Adversarial training (AT) has shown excellent high performance in defending against adversarial examples. Recent studies demonstrate that examples are not equally important to the final robustness of models during AT, that is, the so-called hard examples that can be attacked easily exhibit more influence than robust examples on the final robustness. Therefore, guaranteeing the robustness of hard examples is crucial for improving the final robustness of the model. However, defining effective heuristics to search for hard examples is still difficult. In this article, inspired by the information bottleneck (IB) principle, we uncover that an example with high mutual information of the input and its associated latent representation is more likely to be attacked. Based on this observation, we propose a novel and effective adversarial training method (InfoAT). InfoAT is encouraged to find examples with high mutual information and exploit them efficiently to improve the final robustness of models. Experimental results show that InfoAT achieves the best robustness among different datasets and models in comparison with several state-of-the-art methods.

Masked Modeling-Based Ultrasound Image Classification via Self-Supervised Learning.

Recently, deep learning-based methods have emerged as the preferred approach for ultrasound data analysis. However, these methods often require large-scale annotated datasets for training deep models, which are not readily available in practical scenarios. Additionally, the presence of speckle noise and other imaging artifacts can introduce numerous hard examples for ultrasound data classification. In this paper, drawing inspiration from self-supervised learning techniques, we present a pre-training method based on mask modeling specifically designed for ultrasound data. Our study investigates three different mask modeling strategies: random masking, vertical masking, and horizontal masking. By employing these strategies, our pre-training approach aims to predict the masked portion of the ultrasound images. Notably, our method does not rely on externally labeled data, allowing us to extract representative features without the need for human annotation. Consequently, we can leverage unlabeled datasets for pre-training. Furthermore, to address the challenges posed by hard samples in ultrasound data, we propose a novel hard sample mining strategy. To evaluate the effectiveness of our proposed method, we conduct experiments on two datasets. The experimental results demonstrate that our approach outperforms other state-of-the-art methods in ultrasound image classification. This indicates the superiority of our pre-training method and its ability to extract discriminative features from ultrasound data, even in the presence of hard examples.

Self-Paced Multi-Grained Cross-Modal Interaction Modeling for Referring Expression Comprehension.

As an important and challenging problem in vision-language tasks, referring expression comprehension (REC) generally requires a large amount of multi-grained information of visual and linguistic modalities to realize accurate reasoning. In addition, due to the diversity of visual scenes and the variation of linguistic expressions, some hard examples have much more abundant multi-grained information than others. How to aggregate multi-grained information from different modalities and extract abundant knowledge from hard examples is crucial in the REC task. To address aforementioned challenges, in this paper, we propose a Self-paced Multi-grained Cross-modal Interaction Modeling framework, which improves the language-to-vision localization ability through innovations in network structure and learning mechanism. Concretely, we design a transformer-based multi-grained cross-modal attention, which effectively utilizes the inherent multi-grained information in visual and linguistic encoders. Furthermore, considering the large variance of samples, we propose a self-paced sample informativeness learning to adaptively enhance the network learning for samples containing abundant multi-grained information. The proposed framework significantly outperforms state-of-the-art methods on widely used datasets, such as RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame datasets, demonstrating the effectiveness of our method.

Uncertainty Modeling for Gaze Estimation.

Gaze estimation is an important fundamental task in computer vision and medical research. Existing works have explored various effective paradigms and modules for precisely predicting eye gazes. However, the uncertainty for gaze estimation, e.g., input uncertainty and annotation uncertainty, have been neglected in previous research. Existing models use a deterministic function to estimate the gaze, which cannot reflect the actual situation in gaze estimation. To address this issue, we propose a probabilistic framework for gaze estimation by modeling the input uncertainty and annotation uncertainty. We first utilize probabilistic embeddings to model the input uncertainty, representing the input image as a Gaussian distribution in the embedding space. Based on the input uncertainty modeling, we give an instance-wise uncertainty estimation to measure the confidence of prediction results, which is critical in practical applications. Then, we propose a new label distribution learning method, probabilistic annotations, to model the annotation uncertainty, representing the raw hard labels as Gaussian distributions. In addition, we develop an Embedding Distribution Smoothing (EDS) module and a hard example mining method to improve the consistency between embedding distribution and label distribution. We conduct extensive experiments, demonstrating that the proposed approach achieves significant improvements over baseline and state-of-the-art methods on two widely used benchmark datasets, GazeCapture and MPIIFaceGaze, as well as our collected dataset using mobile devices.

Generalized Parametric Contrastive Learning.

In this paper, we propose the Generalized Parametric Contrastive Learning (GPaCo/PaCo) which works well on both imbalanced and balanced data. Based on theoretical analysis, we observe supervised contrastive loss tends to bias on high-frequency classes and thus increases the difficulty of imbalanced learning. We introduce a set of parametric class-wise learnable centers to rebalance from an optimization perspective. Further, we analyze our GPaCo/PaCo loss under a balanced setting. Our analysis demonstrates that GPaCo/PaCo can adaptively enhance the intensity of pushing samples of the same class close as more samples are pulled together with their corresponding centers and benefit hard example learning. Experiments on long-tailed benchmarks manifest the new state-of-the-art for long-tailed recognition. On full ImageNet, models from CNNs to vision transformers trained with GPaCo loss show better generalization performance and stronger robustness compared with MAE models. Moreover, GPaCo can be applied to semantic segmentation task and obvious improvements are observed on 4 most popular benchmarks. Our code is available at https://github.com/dvlab-research/Parametric-Contrastive-Learning.

YOLO_SRv2: An evolved version of YOLO_SR

Recently, object detection based on deep neural network has become the mainstream research direction of sweeping robots since different obstacles may produce completely contradictory sweeping decisions. However, due to the limited computational ability of the sweeping robot, the size of the object detection network is constrained, which leads to relatively low recognition accuracy and makes it difficult to be applied practically. To address this problem, we present a novel lightweight object detection network, namely YOLO_SRv2, which is an evolved version of our previously proposed You Only Look Once network for Sweeping Robots (YOLO_SR). First, an improved max-pooling structure is presented in the backbone network to avoid losing the suboptimal features. Second, a Convolutional Block Attention Module (CBAM) is added behind the backbone network to enhance the extraction of important features. Finally, a decoupled head module is included to replace the original couple head so as to solve the conflict between classification and regression tasks. In addition, we optimize the classification loss methods to improve the detection performance of hard examples. In the experimental studies, we evaluated the proposed YOLO_SRv2 with other 7 state-of-the-art methods on the ODSR-HIS dataset and PASCAL VOC dataset. The experimental results show that YOLO_SRv2 achieves 86.1% mAP on the ODSR-HIS dataset and 78.2% mAP on PASCAL VOC dataset with 180 FPS on RTX 2080Ti, which verifies the efficiency and the effectiveness of the proposed method as a lightweight object detection network. Source code is athttps://github.com/lvyongshjd/YOLO_SRv2.