Hard Examples Research Articles

OAF-Net: An Occlusion-Aware Anchor-Free Network for Pedestrian Detection in a Crowd

Although pedestrian detection has achieved promising performance with the development of deep learning techniques, it remains a great challenge to detect heavily occluded pedestrians in crowd scenes. Therefore, to make the anchor-free network pay more attention to learning the hard examples of occluded pedestrians, we propose a simple but effective Occlusion-aware Anchor-Free Network (namely OAF-Net) for pedestrian detection in crowd scenes. Specifically, we first design a novel occlusion-aware detection head, which includes three separate center prediction branches combining with the scale and offset prediction branches. In the detection head of OAF-Net, occluded pedestrian instances are assigned to the most suitable center prediction branch according to the occlusion level of human body. To optimize the center prediction, we accordingly propose a novel weighted Focal Loss where pedestrian instances are assigned with different weights according to their visibility ratios, so that the occluded pedestrians are up-weighted during the training process. Our OAF-Net is able to model different occlusion levels of pedestrian instances effectively, and can be optimized towards catching a high-level understanding of the hard training samples of occluded pedestrians. Experiments on the challenging CityPersons, Caltech, and CrowdHuman benchmarks sufficiently validate the efficacy of our OAF-Net for pedestrian detection in crowd scenes.

Physics-informed convolutional neural networks for temperature field prediction of heat source layout without labeled data

Recently, surrogate models based on deep learning have attracted much attention for engineering analysis and optimization. Since constructing data pairs in most engineering problems is time-consuming, data acquisition is becoming the predictive capability bottleneck of most deep surrogate models, which also exist in surrogate for thermal analysis and design. In contrast with data-driven learning, enforcing the physical laws in building surrogates has emerged as a promising alternative to reduce the dependence on annotated data. This paper develops a physics-informed convolutional neural network (CNN) for the thermal simulation surrogate without labeled data. Firstly, we leverage the finite difference method to integrate heat conduction equation and loss function construction, guiding surrogate model training to minimize the violation of physical laws. Since the solution is sensitive to boundary conditions, we properly impose hard constraints by padding in the Dirichlet and Neumann boundaries. The proposed network can learn a mapping from heat source layout to the steady-state temperature field without labeled data, which equals solving an entire family of partial difference equations (PDEs). Moreover, the neural network architecture is well-designed to improve the prediction accuracy of the problem at hand, and pixel-level online hard example mining is proposed to overcome the imbalance of optimization difficulty in the computation domain, which is beneficial to the network training of physics-informed learning. The experiments demonstrate that the proposed method can provide comparable predictions with numerical methods and data-driven deep learning models. We also conduct various ablation studies to investigate the effectiveness of the proposed network components and training methods in this paper. Furthermore, the developed methods can be applied to other design and optimization applications which need to solve parameterized PDEs.

EHHR: an efficient evolutionary hyper-heuristic based recommender framework for short-text classifier selection.

With various machine learning heuristics, it becomes difficult to choose an appropriate heuristic to classify short-text emerging from various social media sources in the form of tweets and reviews. The No Free Lunch theorem asserts that no heuristic applies to all problems indiscriminately. Regardless of their success, the available classifier recommendation algorithms only deal with numeric data. To cater to these limitations, an umbrella classifier recommender must determine the best heuristic for short-text data. This paper presents an efficient reminisce-enabled classifier recommender framework to recommend a heuristic for new short-text data classification. The proposed framework, “Efficient Evolutionary Hyper-heuristic based Recommender Framework for Short-text Classifier Selection (EHHR),” reuses the previous solutions to predict the performance of various heuristics for an unseen problem. The Hybrid Adaptive Genetic Algorithm (HAGA) in EHHR facilitates dataset-level feature optimization and performance prediction. HAGA reveals that the influential features for recommending the best short-text heuristic are the average entropy, mean length of the word string, adjective variation, verb variation II, and average hard examples. The experimental results show that HAGA is 80% more accurate when compared to the standard Genetic Algorithm (GA). Additionally, EHHR clusters datasets and rank heuristics cluster-wise. EHHR clusters 9 out of 10 problems correctly.

Exploring the Limits of Hard Example Mining for ID Document to Selfie Matching

Most face verification systems verify a person’s identity by comparing the ID document with the live face (also called spot face). More specifically, the spot face can be regarded as the probe image and the face in ID document can be regard as the reference image. The identity verification is then conducted by calculating the similarity of the two images. This problem is called ID vs. Spot (IvS) face recognition. IvS face recognition is different from general face recognition, where the IvS face datasets usually contain a very large number of identities (up to millions and more) with only two images per identity. We adopt a metric learning way rather than a classification-based way to train the network on IvS datasets, aiming to avoid tremendous pressure on GPU resource. In this work, we improve the performance mainly from two aspects, namely extending the selecting space to select very hard samples and increasing the diversity of selected samples. For the former, we propose a Super Batch (S-Batch) by aggregating many traditional batches together, in which each anchor sample can select the hard sample pairs from a very large batch. Moreover, we employ a Cross-Batch Hard Example Mining (CBHEM) to select hard samples from not only current batch but also historical batches. For the latter, we propose a Various Batch Sizes (VBS) in our S-Batch, which selects hard samples of different batch scales for training. Extensive experiments demonstrate the effectiveness of our method on IvS face recognition.

SSR-HEF: Crowd Counting With Multiscale Semantic Refining and Hard Example Focusing

Crowd counting based on density maps is generally regarded as a regression task.Deep learning is used to learn the mapping between image content and crowd density distribution. Although great success has been achieved, some pedestrians far away from the camera are difficult to be detected. And the number of hard examples is often larger. Existing methods with simple Euclidean distance algorithm indiscriminately optimize the hard and easy examples so that the densities of hard examples are usually incorrectly predicted to be lower or even zero, which results in large counting errors. To address this problem, we are the first to propose the Hard Example Focusing(HEF) algorithm for the regression task of crowd counting. The HEF algorithm makes our model rapidly focus on hard examples by attenuating the contribution of easy examples.Then higher importance will be given to the hard examples with wrong estimations. Moreover, the scale variations in crowd scenes are large, and the scale annotations are labor-intensive and expensive. By proposing a multi-Scale Semantic Refining (SSR) strategy, lower layers of our model can break through the limitation of deep learning to capture semantic features of different scales to sufficiently deal with the scale variation. We perform extensive experiments on six benchmark datasets to verify the proposed method. Results indicate the superiority of our proposed method over the state-of-the-art methods. Moreover, our designed model is smaller and faster.

Deep supervised hashing with hard example pairs optimization for image retrieval

Deep supervised hashing with hard example pairs optimization for image retrieval

A Seam Tracking Method Based on an Image Segmentation Deep Convolutional Neural Network

Vision-based welding seam tracking is an important and unique branch of welding automation. Active vision seam tracking systems achieve accurate feature extraction by using an auxiliary light source, but this will introduce extra costs and the real-time performance will be affected. In contrast, passive vision systems achieve better real-time performance and their structure is relatively simple. This paper proposes a passive vision welding seam tracking system in Plasma Arc Welding (PAW) based on semantic segmentation. The BiseNetV2 network is adopted in this paper and online hard example mining (OHEM) is used to improve the segmentation effect. This network structure is a lightweight structure allowing effective image feature extraction. According to the segmentation results, the offset between the welding seam and the welding torch can be calculated. The results of the experiments show that the proposed method can achieve 57 FPS and the average error of the offset calculation is within 0.07 mm, meaning it can be used for real-time seam tracking.

Distributionally Robust Deep Learning using Hardness Weighted Sampling

Limiting failures of machine learning systems is of paramount importance for safety-critical applications. In order to improve the robustness of machine learning systems, Distributionally Robust Optimization (DRO) has been proposed as a generalization of Empirical Risk Minimization (ERM). However, its use in deep learning has been severely restricted due to the relative inefficiency of the optimizers available for DRO in comparison to the wide-spread variants of Stochastic Gradient Descent (SGD) optimizers for ERM.<br>We propose SGD with hardness weighted sampling, a principled and efficient optimization method for DRO in machine learning that is particularly suited in the context of deep learning. Similar to a hard example mining strategy in practice, the proposed algorithm is straightforward to implement and computationally as efficient as SGD-based optimizers used for deep learning, requiring minimal overhead computation. In contrast to typical ad hoc hard mining approaches, we prove the convergence of our DRO algorithm for over-parameterized deep learning networks with ReLU activation and finite number of layers and parameters.<br>Our experiments on fetal brain 3D MRI segmentation and brain tumor segmentation in MRI demonstrate the feasibility and the usefulness of our approach. Using our hardness weighted sampling for training a state-of-the-art deep learning pipeline leads to improved robustness to anatomical variabilities in automatic fetal brain 3D MRI segmentation using deep learning and to improved robustness to the image protocol variations in brain tumor segmentation.a decrease of 2% of the interquartile range of the Dice scores for the enhanced tumor and the tumor core regions.<br>Our code is available at <a href='https://github.com/LucasFidon/HardnessWeightedSampler'>https://github.com/LucasFidon/HardnessWeightedSampler</a>

AANet: an attention-based alignment semantic segmentation network for high spatial resolution remote sensing images

ABSTRACT In this paper, we present an efficient network to tackle three critical problems in high spatial resolution (HSR) remote sensing image segmentation: feature misalignment, insufficient contextual information extraction and various class imbalance issues. In detail, we propose a novel Feature Alignment Block (FAB) to suppress misalignment issues with the guide of an anchor map. Further, to extract sufficient information, we design a Contextual Augmentation Block (CAB) to augment features of different semantic levels. Finally, we present an Annealing Online Hard Example Mining (AOHEM) strategy to handle the various class imbalance issues with a view to dynamically adjust the focus of the network. We apply the above proposed designs to FPN to form our Attention-based Alignment Network (AANet). Experimental results demonstrate that the proposed method achieves promising results on the challenging iSAID and Vaihingen datasets with a better trade-off between accuracy and complexity.

An End-to-End Multi-Task Learning Model for Drivable Road Detection via Edge Refinement and Geometric Deformation

This paper presents a road detection method for autonomous driving based on an end-to-end neural network model. Our method takes advantage of both the characteristics of road boundary and multi-task learning of a deep convolutional network. By reassigning the label and rebalancing the loss of road pixels, we focus on the learning of hard examples on the boundary to refine its performance. Then, a road geometric transformation-based data augmentation method is proposed to enable the network model to be robust under traffic scenes. Based on these two novel methods, a unified architecture consisting of a shared deep residual encoder network and multi-branch decoder sub-networks is integrated. It adopts road scene classification as a supervised learning task to realize road segmentation and scene classification simultaneously. Experiments demonstrate that the proposed method has achieved the highest MaxF value in most road scenes. Both qualitative and quantitative evaluations on the KITTI-Road benchmark demonstrate our superior performance.

Semi-supervised Learning for Weed and Crop Segmentation Using UAV Imagery.

Weed control has received great attention due to its significant influence on crop yield and food production. Accurate mapping of crop and weed is a prerequisite for the development of an automatic weed management system. In this paper, we propose a weed and crop segmentation method, SemiWeedNet, to accurately identify the weed with varying size in complex environment, where semi-supervised learning is employed to reduce the requirement of a large amount of labelled data. SemiWeedNet takes the labelled and unlabelled images into account when generating a unified semi-supervised architecture based on semantic segmentation model. A multiscale enhancement module is created by integrating the encoded feature with the selective kernel attention, to highlight the significant features of the weed and crop while alleviating the influence of complex background. To address the problem caused by the similarity and overlapping between crop and weed, an online hard example mining (OHEM) is introduced to refine the labelled data training. This forces the model to focus more on pixels that are not easily distinguished, and thus effectively improve the image segmentation. To further exploit the meaningful information of unlabelled data, consistency regularisation is introduced by maintaining the context consistency during training, making the representations robust to the varying environment. Comparative experiments are conducted on a publicly available dataset. The results show the SemiWeedNet outperforms the state-of-the-art methods, and its components have promising potential in improving segmentation.

Iterative Feature Matching for Self-Supervised Indoor Depth Estimation

In this paper, we propose an iterative feature matching framework for self-supervised depth estimation in indoor scenes. Conventional methods usually leverage the structure-from-motion supervision to help the photometric optimization escape from the local minima, which have complex ego-motion and large regions with non-texture or repeated-texture. However, the supervision is limited as the reconstruction is usually sparse. To address this, we propose an iterative feature matching framework called IFMNet to jointly learn depths and search for correspondences. With the predicted depths from the previous iteration, we present an online optimized grid searching algorithm to find more accurate correspondences. Given these new correspondences, we compute the triangulated depths and improve the depth network with adaptive bin-wise online hard example mining. Experimental results on the NYU Depth V2 and SceneNet datasets verify the effectiveness of our approach.

THE MONITORING OF NON-FARMING AND NON-GRAIN PURPOSES IN ARABLE LAND OF ZHEJIANG, CHINA WITH DOMESTIC SATELLITE IMAGERY

Abstract. Arable land protection is essential for Zero hunger the Sustainable Development Goals of United Nations, and the arable land protection includes two aspects, non-farming and non-grain. We try to monitor the arable land protection in Zhejiang with domestic satellite imagery. Satellite remote technology has become an essential way to monitor the land cover change (for non-farming) and grain crops (for non-grain). However, current monitoring frequency and scale were unable to satisfy the needs for non-farming monitoring. The low-resolution image cannot satisfy the feature of land fragmentation of Zhejiang for non-grain monitoring. To address the above problem, this paper proposes a land cover change method to monitor non-farming purposes based on deeplabv3+ with monthly coverage 2 meters resolution images. By focusing on rebuilding training data set and improving training strategy with hard example training, the difficulty of the spurious change caused by the adjustment of farming structure is solved. At the same time, this paper builds three training processes (Initial training, Fine training, Retraining for promotion) based on Fully Convolutional Neural Network FCN-8S to monitor the main grain crops in Zhejiang. The phenological features are added into the process of training to further improve the accuracy. At present, land cover change method of this paper has been applied in Zhejiang province and the monitoring of grain crops has been carried out in some regions according to the specific requirements. The result shows that both the two methods exhibit good accuracy and generalization ability at the time and space scale.

Automatic concrete crack segmentation model based on transformer

Routine visual inspection of concrete structures is essential to maintain safe conditions. Therefore, studies of concrete crack segmentation using deep learning methods have been extensively conducted in recent years. However, insufficient performance remains a major challenge in diverse field-inspection scenarios. In this study, a novel SegCrack model for pixel-level crack segmentation is therefore proposed using a hierarchically structured Transformer encoder to output multiscale features and a top-down pathway with lateral connections to progressively up-sample and fuse features from the deepest layer of the encoder. Furthermore, an online hard example mining strategy was adopted to strengthen the detection of hard samples and improve the model performance. The effect of dataset size on the segmentation performance was then investigated. The results indicated that SegCrack achieved a precision, recall, F1 score, and mean intersection over union of 96.66%, 95.46%, 96.05%, and 92.63%, respectively, using the test set.

SFGNet detecting objects via spatial fine-grained feature and enhanced RPN with spatial context

Object detection, which is one of the most fundamental visual recognition tasks, has been a hotspot in computer vision. CNN (Convolutional Neural Networks) have been widely employed for building detector. Due to the success of RPN (Region Proposal Network), the two-stage detectors get both classification accuracy and precise regression bounding boxes. However, they still struggle in small-size object detection. In this paper, we present a deep network, namely Spatial Fine-Grained Network (SFGN). The SFGN that exploits Spatial Fine-Grained Features (SFGF) concatenates the higher resolution features, which is fine-grained with the low resolution features and high-level semantic by stacking spatial features for fine-grained features. An enhanced region proposal generator is proposed to get the objectless for small object to obtain a small set of proposal. The contextual information surrounding the region of interest is embedded using local spatial information for increasing the useful information and discriminating the background. For improving the detection performance, we use a simple yet surprisingly effective online hard example mining (OHEM) algorithm for training region proposal generator. It embeds an efficiently implemented soft non-maximum suppression (soft-NMS) for replacing with tradition NMS to obtain consistent improvements without increasing the computational complexity in inference. On PASCAL VOC 2007 and PASCAL VOC 2012 datasets, our SFGN improves baseline model from 81.2% mAP to 80.6% mAP. On MS COCO dataset, SFGN also performs better than baseline model. As intuition suggests, our detection results provide strong evidence that our SFGN improves detection accuracy, especially in small object test.

Fault Diagnosis Method of Distribution Equipment Based on Hybrid Model of Robot and Deep Learning

In view of the poor effect of most fault diagnosis methods on the intelligent recognition of equipment images, a fault diagnosis method of distribution equipment based on the hybrid model of robot and deep learning is proposed to reduce the dependence on manpower and realize efficient intelligent diagnosis. Firstly, the robot is used to collect the on-site state images of distribution equipment to build the image information database of distribution equipment. At the same time, the robot background is used as the comprehensive database data analysis platform to optimize the sample quality of the database. Then, the massive infrared images are segmented based on chroma saturation brightness space to distinguish the defective equipment images, and the defective equipment areas are extracted from the images by OTSU method. Finally, the residual network is used to improve the region-based fully convolutional networks (R-FCN) algorithm, and the improved R-FCN algorithm trained by the online hard example mining method is used for fault feature learning. The fault type, grade, and location of distribution equipment are obtained through fault criterion analysis. The experimental analysis of the proposed method based on PyTorch platform shows that the fault diagnosis time and accuracy are about 5.5 s and 92.06%, respectively, which are better than other comparison methods and provide a certain theoretical basis for the automatic diagnosis of power grid equipment.

Two-stage streaming keyword detection and localization with multi-scale depthwise temporal convolution

A keyword spotting (KWS) system running on smart devices should accurately detect the appearances and predict the locations of predefined keywords from audio streams, with small footprint and high efficiency. To this end, this paper proposes a new two-stage KWS method which combines a novel multi-scale depthwise temporal convolution (MDTC) feature extractor and a two-stage keyword detection and localization module. The MDTC feature extractor learns multi-scale feature representation efficiently with dilated depthwise temporal convolution, modeling both the temporal context and the speech rate variation. We use a region proposal network (RPN) as the first-stage KWS. At each frame, we design multiple time regions, which all take the current frame as the end position but have different start positions. These time regions (or formally anchors) are used to indicate rough location candidates of keyword. With frame level features from the MDTC feature extractor as inputs, RPN learns to propose keyword region proposals based on the designed anchors. To alleviate the keyword/non-keyword class imbalance problem, we specifically introduce a hard example mining algorithm to select effective negative anchors in RPN training. The keyword region proposals from the first-stage RPN contain keyword location information which is subsequently used to explicitly extract keyword related sequential features to train the second-stage KWS. The second-stage system learns to classify and transform region proposal to keyword IDs and ground-truth keyword region respectively. Experiments on the Google Speech Command dataset show that the proposed MDTC feature extractor surpasses several competitive feature extractors with a new state-of-the-art command classification error rate of 1.74%. With the MDTC feature extractor, we further conduct wake-up word (WuW) detection and localization experiments on a commercial WuW dataset. Compared to a strong baseline, our proposed two-stage method achieves relatively 27–32% better false rejection rate at one false alarm per hour, while for keyword localization, the two-stage approach achieves more than 0.95 mean intersection-over-union ratio, which is clearly better than the one-stage RPN method.

An Improved Cervical Cell Segmentation Method Based on Deep Convolutional Network

The cervical cytology smear test is an effective method for cervical cancer early screening, and segmentation accuracy is essential for computer-aided diagnosis. In this study, an improved cervical nucleus and cytoplasm segmentation method based on a deep convolutional network was proposed. This method consisted of a cellular region proposal and pixel-level segmentation network (CRP-PSN). Data were obtained from the 2014 International Symposium on Biomedical Imaging cervical cell segmentation competition open dataset. In CRP networks, online hard example mining and soft-nonmaximum suppression algorithms were incorporated to solve problems, including background noise, impurities, and other interferences in smear images. For PSN networks, a generative adversarial network-generated adversarial network algorithm and least-squares loss function were used to generate cell region segmentation, thereby improving the cytoplasm segmentation results. Finally, the improved CRP-PSN model was analyzed and compared with other typical cervical cell segmentation methods. The experimental results showed that the proposed model could effectively improve the segmentation accuracy of cytoplasm and nuclei in cervical cytology smear images by 92% and 98.6%, respectively. These findings provided strong support for the application of this method for automated interpretation of cervical cytology smear images and improved diagnostic reliability.

Soft pseudo-Label shrinkage for unsupervised domain adaptive person re-identification

One effective way to tackle unsupervised domain adaptation (UDA) on person re-identification (Re-ID) is to use clustering-based self-training approach, where a model is trained with hard pseudo-labels obtained from a clustering method. Using a hard pseudo-label, a sample is assigned to the cluster with the highest probability, which is sensitive to the incorrect clustering result due to imperfect clustering algorithms. Soft pseudo-labels can mitigate this issue by representing the sample with the full range of class probabilities from all clusters. Specifically, soft pseudo-labels comprise probabilities of full range classes, because they consider both the hard samples and easy samples. This will distract the model from learning more discriminative features in the hard examples. To solve this issue, we propose a coarse-to-fine refinement mechanism to produce robust refined soft pseudo-labels by progressively focusing more on the hard samples while less on the easy samples. The proposed refined soft pseudo-labels can be readily integrated into cross-entropy loss as a strong supervision to guide the model to learn more discriminative features. Extensive experiments demonstrate that our proposed method outperforms the state-of-the-art unsupervised domain adaptation approaches on person Re-ID with a considerable margin. Code will be available at: http://github.com/Dingyuan-Zheng/ctf-UDA.

DBFU-Net: Double branch fusion U-Net with hard example weighting train strategy to segment retinal vessel.

BackgroundMany fundus imaging modalities measure ocular changes. Automatic retinal vessel segmentation (RVS) is a significant fundus image-based method for the diagnosis of ophthalmologic diseases. However, precise vessel segmentation is a challenging task when detecting micro-changes in fundus images, e.g., tiny vessels, vessel edges, vessel lesions and optic disc edges.MethodsIn this paper, we will introduce a novel double branch fusion U-Net model that allows one of the branches to be trained by a weighting scheme that emphasizes harder examples to improve the overall segmentation performance. A new mask, we call a hard example mask, is needed for those examples that include a weighting strategy that is different from other methods. The method we propose extracts the hard example mask by morphology, meaning that the hard example mask does not need any rough segmentation model. To alleviate overfitting, we propose a random channel attention mechanism that is better than the drop-out method or the L2-regularization method in RVS.ResultsWe have verified the proposed approach on the DRIVE, STARE and CHASE datasets to quantify the performance metrics. Compared to other existing approaches, using those dataset platforms, the proposed approach has competitive performance metrics. (DRIVE: F1-Score = 0.8289, G-Mean = 0.8995, AUC = 0.9811; STARE: F1-Score = 0.8501, G-Mean = 0.9198, AUC = 0.9892; CHASE: F1-Score = 0.8375, G-Mean = 0.9138, AUC = 0.9879).DiscussionThe segmentation results showed that DBFU-Net with RCA achieves competitive performance in three RVS datasets. Additionally, the proposed morphological-based extraction method for hard examples can reduce the computational cost. Finally, the random channel attention mechanism proposed in this paper has proven to be more effective than other regularization methods in the RVS task.