Hard Examples Research Articles

The Pandora Papers Opens up Pandora’s Box: Integrity in Crisis

Integrity lies at the center of state-society interactions, with broad implications for the practice of public administration, governance, and public service delivery. The Pandora Papers leaks are hard examples of integrity violations, shining a light on how public administration operates in practice. The integrity deficits revealed not only threaten people’s wellbeing, but also put on display the state’s leniency toward rich and powerful people accumulating wealth through corruption, fraud, and tax evasion, among other crimes. In this context, this article attempts to understand whether the Pandora Papers leaks violate of the fundamentals of integrity. Drawing on cases and examples, the findings of the article show that the Pandora Papers illustrate the integrity deficit of statesmen, political leaders, businesspersons, and bureaucrats, the people responsible for ensuring integrity in managing the state and statecraft. The article further elucidates that bad governance affects integrity, both nationally and globally.

Damaged Building Extraction Using Modified Mask R-CNN Model Using Post-Event Aerial Images of the 2016 Kumamoto Earthquake

Remote sensing is an effective method of evaluating building damage after a large-scale natural disaster, such as an earthquake or a typhoon. In recent years, with the development of computer vision technology, deep learning algorithms have been used for damage assessment from aerial images. In April 2016, a series of earthquakes hit the Kyushu region, Japan, and caused severe damage in the Kumamoto and Oita Prefectures. Numerous buildings collapsed because of the strong and continuous shaking. In this study, a deep learning model called Mask R-CNN was modified to extract residential buildings and estimate their damage levels from post-event aerial images. Our Mask R-CNN model employs an improved feature pyramid network and online hard example mining. Furthermore, a non-maximum suppression algorithm across multiple classes was also applied to improve prediction. The aerial images captured on 29 April 2016 (two weeks after the main shock) in Mashiki Town, Kumamoto Prefecture, were used as the training and test sets. Compared with the field survey results, our model achieved approximately 95% accuracy for building extraction and over 92% accuracy for the detection of severely damaged buildings. The overall classification accuracy for the four damage classes was approximately 88%, demonstrating acceptable performance.

Distorted dense analogs intelligent recognition in fisheye lenses by partially featured points calibrating and clustering.

Dense analogs intelligent recognition (DAIR) has many potential applications in various fields as a new cross-disciplinary frontier of artificial intelligence and optical technology. However, with extensive application of fisheye lenses, inherent distortions in fisheye images have brought new challenges to DAIR. To solve this problem, we propose and experimentally demonstrate a partially featured points calibrating method that needs only correction of central points of the bounding boxes output by a convolutional neural network (CNN). The key to our method is a central-coordinate calibrating and clustering algorithm (CCCCA) based on a hemispheric double longitude projection model. Experimental results show that the CCCCA reduces the classification error rate by 6.05%, enhancing the classification accuracy of distorted DAIR up to 99.31%. Such classification accuracy is about 2.74% higher than that achieved by the mainstream online hard example mining algorithm, effectively modifying recognition errors induced by the CNN.

QuadNet: Quadruplet loss for multi-view learning in baggage re-identification

Recently, baggage re-identification (ReID) has become an attractive topic in computer vision because it plays an important role in intelligent surveillance. However, the wide variations in different views of baggage items degrade baggage ReID performance. In this paper, a novel QuadNet is proposed to solve the multi-view problem in baggage ReID at three levels. At the sample level, we propose a multi-view sampling strategy which samples hard examples from multiple identities in multiple views. The sampled baggage items are used to construct quadruplets. At the feature level, view-aware attentional local features are extracted from discriminative regions in each view. These local features are fused with global features to obtain better representations of the quadruplets. At the loss level, a multi-view quadruplet loss operating on the representations of quadruplets is proposed to reduce the intra-class distances caused by view variations and increase the inter-class distances of baggage images captured in the same view. A random local blur data augmentation is proposed to handle the motion blur which is often found in baggage images. The multi-task learning of materials is introduced to obtain discriminative features based on the materials of baggage surfaces. Extensive experiments on three ReID datasets, MVB, Market-1501 and VeRi-776, indicate the remarkable effectiveness and good generalization of the QuadNet model. It has achieved the state-of-the-art performance on the three datasets.

Soft focal loss: Evaluating sample quality for dense object detection

Classic detectors divide the candidate boxes into positive and negative groups based on their intersection-over-union (IoU) with matched objects. Such a sharp label assignment method does not directly consider the distance between the centers of the two boxes. Consequently, some positive samples are ambiguous, which limits the detection performance. In this paper, we propose a new label assignment method by considering two different perspectives. The first perspective is IoU to indicate the degree of overlap. The other one is the defined variable to characterize the center-distance between the candidate box and its matched ground truth. In addition, the classification is usually optimized by Focal Loss for paying more attention to hard examples, but it affects the training of high-quality samples. Therefore, we define the Soft Focal Loss (SFL) and the quality factor that reflects the quality of the samples. Embedding the quality factor into SFL makes the network focus on learning high-quality rather than hard examples. Furthermore, the quality factor is utilized to re-weight the classification and regression losses to enhance the correlation between these two tasks. Experiments on COCO show that the proposed approach can improve RetinaNet by 1.3% and 1.2% AP with backbone ResNet-50 and ResNet-101 in 1x training schedule, without incurring any additional overhead.

Knowing What to Learn: A Metric-Oriented Focal Mechanism for Image Captioning.

Despite considerable progress, image captioning still suffers from the huge difference in quality between easy and hard examples, which is left unexploited in existing methods. To address this issue, we explore the hard example mining in image captioning, and propose a simple yet effective mechanism to instruct the model to pay more attention to hard examples, thereby improving the performance in both general and complex scenarios. We first propose a novel learning strategy, termed Metric-oriented Focal Mechanism (MFM), for hard example mining in image captioning. Differing from the existing strategies for classification tasks, MFM can adopt the generative metrics of image captioning to measure the difficulties of examples, and then up-weight the rewards of hard examples during training. To make MFM applicable to different datasets without tedious parameter tuning, we further introduce an adaptive reward metric called Effective CIDEr (ECIDEr), which considers the data distribution of easy and hard examples during reward estimation. Extensive experiments are conducted on the MS COCO benchmark, and the results show that while maintaining the performance on simple examples, MFM can significantly improve the quality of captions for hard examples. The ECIDEr-based MFM is equipped on the current SOTA method, e.g., DLCT (Luo et al., 2021), which outperforms all existing methods and achieves new state-of-the-art performance on both the off-line and on- line testing, i.e., 134.3 CIDEr for the off-line testing and 136.1 for the on- line testing of MSCOCO. To validate the generalization ability of ECIDEr-based MFM, we also apply it to another dataset, namely Flickr30k, and superior performance gains can also be obtained.

Cross-Batch Hard Example Mining With Pseudo Large Batch for ID vs. Spot Face Recognition.

In our daily life, a large number of activities require identity verification, e.g., ePassport gates. Most of those verification systems recognize who you are by matching the ID document photo (ID face) to your live face image (spot face). The ID vs. Spot (IvS) face recognition is different from general face recognition where each dataset usually contains a small number of subjects and sufficient images for each subject. In IvS face recognition, the datasets usually contain massive class numbers (million or more) while each class only has two image samples (one ID face and one spot face), which makes it very challenging to train an effective model (e.g., excessive demand on GPU memory if conducting the classification on such massive classes, hardly capture the effective features for bisample data of each identity, etc.). To avoid the excessive demand on GPU memory, a two-stage training method is developed, where we first train the model on the dataset in general face recognition (e.g., MS-Celeb-1M) and then employ the metric learning losses (e.g., triplet and quadruplet losses) to learn the features on IvS data with million classes. To extract more effective features for IvS face recognition, we propose two novel algorithms to enhance the network by selecting harder samples for training. Firstly, a Cross-Batch Hard Example Mining (CB-HEM) is proposed to select the hard triplets from not only the current mini-batch but also past dozens of mini-batches (for convenience, we use batch to denote a mini-batch in the following), which can significantly expand the space of sample selection. Secondly, a Pseudo Large Batch (PLB) is proposed to virtually increase the batch size with a fixed GPU memory. The proposed PLB and CB-HEM can be employed simultaneously to train the network, which dramatically expands the selecting space by hundreds of times, where the very hard sample pairs especially the hard negative pairs can be selected for training to enhance the discriminative capability. Extensive comparative evaluations conducted on multiple IvS benchmarks demonstrate the effectiveness of the proposed method.

Self-Knowledge Distillation Based on Knowledge Transfer from Soft to Hard Examples

Self-Knowledge Distillation Based on Knowledge Transfer from Soft to Hard Examples

$\text{A}^{3}$Seg: An Annealing Augmented and Aligned Semantic Segmentation Network for High Spatial Resolution Remote Sensing Images

Semantic segmentation of high spatial resolution (HSR) remote sensing images (RSIs) plays an important role in many applications. However, HSR RSIs have significantly larger spatial sizes than typical natural images, which results in fewer valuable samples when training models. In addition, fusing multiscale features is the key step in obtaining features with strong semantic and high spatial information. However, current feature fusion methods are too straightforward to address misalignment issues. To handle these two problems, we propose an annealing augmented and aligned segmentation network, named <inline-formula><tex-math notation="LaTeX">$\text{A}^{3}$</tex-math></inline-formula>Seg. Specifically, we propose an annealing online hard example mining (AOHEM) strategy to automatically select more valuable samples during the training stage. Based on AOHEM, a contextual augmentation block is proposed to extract sufficient contextual information using three different attention mechanisms in consideration of three different feature properties. Finally, we propose a novel feature alignment block to fuse features at different levels by alignment with the guidance of a salient feature. Experimental results on three different HSR RSIs datasets demonstrate that the proposed method outperforms the state-of-the-art general semantic segmentation methods with a better tradeoff between accuracy and complexity.

Universal Adversarial Patch Attack for Automatic Checkout Using Perceptual and Attentional Bias.

Adversarial examples are inputs with imperceptible perturbations that easily mislead deep neural networks (DNNs). Recently, adversarial patch, with noise confined to a small and localized patch, has emerged for its easy feasibility in real-world scenarios. However, existing strategies failed to generate adversarial patches with strong generalization ability due to the ignorance of the inherent biases of models. In other words, the adversarial patches are always input-specific and fail to attack images from all classes or different models, especially unseen classes and black-box models. To address the problem, this paper proposes a bias-based framework to generate universal adversarial patches with strong generalization ability, which exploits the perceptual bias and attentional bias to improve the attacking ability. Regarding the perceptual bias, since DNNs are strongly biased towards textures, we exploit the hard examples which convey strong model uncertainties and extract a textural patch prior from them by adopting the style similarities. The patch prior is closer to decision boundaries and would promote attacks across classes. As for the attentional bias, motivated by the fact that different models share similar attention patterns towards the same image, we exploit this bias by confusing the model-shared similar attention patterns. Thus, the generated adversarial patches can obtain stronger transferability among different models. Taking Automatic Check-out (ACO) as the typical scenario, extensive experiments including white-box/black-box settings in both digital-world (RPC, the largest ACO related dataset) and physical-world scenario (Taobao and JD, the world's largest online shopping platforms) are conducted. Experimental results demonstrate that our proposed framework outperforms state-of-the-art adversarial patch attack methods.

Cascaded Multi-Task Road Extraction Network for Road Surface, Centerline, and Edge Extraction

Road extraction from very high-resolution (VHR) remote sensing imagery remains a huge challenge, due to the shadows and occlusions of trees and buildings. Such complex backgrounds result in deep networks often producing fragmented roads with poor connectivity. Road extraction has three typical tasks: road surface segmentation (SS), centerline extraction (CE), and edge detection (ED), which are conducted in a wide range of real applications. Also, the three tasks have a symbiotic relationship, i.e., the road SS determines the location of the centerline and edges, and the CE and ED can allow the generation of more continuous road surfaces. However, most of the previous works have completed these three tasks separately, without exploiting the symbiotic relationship between them to boost the road connectivity. In this article, in order to improve road connectivity, a cascaded multitask (CasMT) road extraction framework for simultaneously extracting the road surface, centerline, and edges is proposed. In the proposed framework, topology-aware learning is applied to capture the long-distance topological relationships, and hard example mining (HEM) loss is employed to focus more on hard samples, to further enhance the road completeness. Extensive experiments were conducted on the DeepGlobe road dataset and a large-scale road dataset (called the LSCC dataset) from the three Chinese cities of Beijing, Shanghai, and Wuhan. The experimental results obtained on the public DeepGlobe dataset demonstrate that the proposed CasMT framework can significantly outperform the current state-of-the-art method. Moreover, the generalization capability of the model was verified on the LSCC dataset, where the proposed CasMT framework achieved the best performance in the average path length similarity (APLS) road topology metric, which further confirms the superiority of the proposed framework.

Prototype Memory for Large-Scale Face Representation Learning

Face representation learning using datasets with a massive number of identities requires appropriate training methods. Softmax-based approach, currently the state-of-the-art in face recognition, in its usual "full softmax" form is not suitable for datasets with millions of persons. Several methods, based on the "sampled softmax" approach, were proposed to remove this limitation. These methods, however, have a set of disadvantages. One of them is a problem of "prototype obsolescence": classifier weights (prototypes) of the rarely sampled classes receive too scarce gradients and become outdated and detached from the current encoder state, resulting in incorrect training signals. This problem is especially serious in ultra-large-scale datasets. In this paper, we propose a novel face representation learning model called Prototype Memory, which alleviates this problem and allows training on a dataset of any size. Prototype Memory consists of the limited-size memory module for storing recent class prototypes and employs a set of algorithms to update it in appropriate way. New class prototypes are generated on the fly using exemplar embeddings in the current mini-batch. These prototypes are enqueued to the memory and used in a role of classifier weights for softmax classification-based training. To prevent obsolescence and keep the memory in close connection with the encoder, prototypes are regularly refreshed, and oldest ones are dequeued and disposed of. Prototype Memory is computationally efficient and independent of dataset size. It can be used with various loss functions, hard example mining algorithms and encoder architectures. We prove the effectiveness of the proposed model by extensive experiments on popular face recognition benchmarks.

The Outcome of the 2022 Landslide4Sense Competition: Advanced Landslide Detection From Multisource Satellite Imagery

The scientific outcomes of the 2022 Landslide4Sense (L4S) competition organized by the Institute of Advanced Research in Artificial Intelligence (IARAI) are presented here. The objective of the competition is to automatically detect landslides based on large-scale multiple sources of satellite imagery collected globally. The 2022 L4S aims to foster interdisciplinary research on recent developments in deep learning (DL) models for the semantic segmentation task using satellite imagery. In the past few years, DL-based models have achieved performance that meets expectations on image interpretation, due to the development of convolutional neural networks (CNNs). The main objective of this article is to present the details and the best-performing algorithms featured in this competition. The winning solutions are elaborated with state-of-the-art models like the Swin Transformer, SegFormer, and U-Net. Advanced machine learning techniques and strategies such as hard example mining, self-training, and mix-up data augmentation are also considered. Moreover, we describe the L4S benchmark data set in order to facilitate further comparisons, and report the results of the accuracy assessment online. The data is accessible on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Future Development Leaderboard</i> for future evaluation at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://www.iarai.ac.at/landslide4sense/challenge/</uri> , and researchers are invited to submit more prediction results, evaluate the accuracy of their methods, compare them with those of other users, and, ideally, improve the landslide detection results reported in this article.

Boosted Mask R-CNN algorithm for accurately detecting strawberry plant canopies in the fields from low-altitude drone images

The research constructs a novel structure by integrating two parts of online object detection pipelines into the current state-of-the-art Mask R-CNN algorithm to improve the detection performance. The DJI Mavic Air drone is used to collect low-altitude sensing images of strawberry plant canopies. The data augmentation method is employed to feed more instances into the original image dataset to boost the generalization and robustness of the detection model in the training procedure. A ResNet50 backbone combined with a feature pyramid network is presented to extract the features of strawberry plant canopies. The online hard example mining algorithm is introduced to mine hard samples to learn rich features and update model weights. Soft non-maximum suppression based on recursive application on the remaining detection boxes within the predefined overlap threshold is proposed to improve the performance of identifying the large complex overlapping area of strawberry plant canopies. The qualitative results demonstrate that the improved detection model had an AP50 of 96.9 and an AR of 78.5 on the test set, which are approximately 30% higher than the original values.

An SAR Target Detector Based on Gradient Harmonized Mechanism and Attention Mechanism

In this letter, a target detector based on gradient harmonized mechanism (GHM) and attention mechanism is proposed to realize synthetic aperture radar (SAR) target detection in complex scenes. Considering the imbalance of positive and negative examples in SAR target detection, we use RefineDet as our backbone network. RefineDet can mitigate this imbalance problem by introducing the idea of two-step classification and regression into the one-stage detector. However, RefineDet only selects a part of examples for training and does not make full use of the information of all examples. Therefore, we apply GHM to the classification loss function of RefineDet, so that the network can make full use of all examples and increase the weights of hard examples adaptively in the loss function to reduce the false alarms and the missing alarms. In addition, to achieve a better detection performance in SAR images with complex scenes, a multiscale feature attention module (MFAM) is embedded into the network. By applying channel and spatial attention mechanisms to the multiscale feature maps, the MFAM can highlight the significant information and suppress the interference caused by clutter. The extensive experimental results based on the measured SAR dataset verify the effectiveness of the proposed method.

Two-Stage Underwater Object Detection Network Using Swin Transformer

Underwater object detection plays an essential role in ocean exploration, and the increasing amount of underwater object image data makes the study of advanced underwater object detection algorithms of great practical significance. However, there are problems with colour offset, low contrast, and target blur in underwater image data. An underwater object detection algorithm based on Faster R-CNN is proposed to solve these problems. First, the Swin Transformer is used as the backbone network of the algorithm. Second, by adding the path aggregation network, the deep feature map and the shallow feature map are superimposed and fused. Third, online hard example mining, makes the training process more efficient. Fourth, the ROI pooling is improved to ROI align, eliminating the two quantization errors of ROI pooling and improving the detection performance. Compared with other algorithms, the proposed algorithm’s based on improved Faster-RCNN on URPC2018 dataset is improved to 80.54%, and basically solve the problem of missed detection and false detection of objects of different sizes in a complex environment.

Calibrated Focal Loss for Semantic Labeling of High-Resolution Remote Sensing Images

Currently, the most advanced high-resolution remote sensing image (HRRSI) semantic labeling methods rely on deep neural networks. However, HRRSIs naturally have a serious class imbalance problem which is not yet well solved by the current method. The cross-entropy (CE) loss is often used to guide the training of semantic labeling neural networks for HRRSIs, but it is essentially dominated by the major classes in the image, resulting in poor predictions for the minority class. Based on the prediction results, Focal Loss (FL) effectively suppresses the negative impact of class imbalance in dense object detection by redistributing the loss of each sample. In this paper, we thoroughly analyze the inadequacy of FL for semantic labeling, which inevitably introduces confusing-classified examples that are more difficult to classify while suppressing the loss of well-classified examples. Therefore, following the core idea of FL, we redefine the hard examples in semantic labeling of HRRSIs and propose the prediction confusion map (PCM) to measure the classification difficulty. Based on this, we further propose the Calibrated Focal Loss (CFL) for semantic labeling of HRRSIs. Finally, we conduct complete experiments on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam datasets to analyze the semantic labeling performance, model uncertainty, and confidence calibration of different loss functions. Experimental results show that CFL can achieve outstanding results compared to other commonly used loss functions without increasing model parameters and training iterations, demonstrating the effectiveness of our method. In the end, combined with our previously proposed HCANet, we further verify the effectiveness of CFL on state-of-the-art network structures.

Geological Remote Sensing Interpretation Using Deep Learning Feature and an Adaptive Multisource Data Fusion Network

Geological remote sensing interpretation can extract elements of interest from multiple types of images, which is vital in geological survey and mapping, especially in inaccessible regions. However, due to numerous classes, high interclass similarities, complex distributions, and sample imbalances of geological elements, the interpretation results of machine-learning (ML)-based methods are understandably worse than manual visual interpretation. Additionally, scholars in remote sensing have mainly carried out their works to interpret a single geological element category, such as mineral, lithological, soil and structure. The interpretation of multiple geological elements is missing, which is more in line with the open world. To improve the interpretation results of ML-based methods and reduce the labor cost in geological survey and mapping, we propose a deep-learning (DL)-feature-based adaptive multi-source data fusion network (AMSDFNet) for the efficient interpretation of multiple geological remote sensing elements. The AMSDFNet has two branches for learning valuable spatial and spectral information from two kinds of data sources, wherein the atrous spatial pyramid pooling operation and an attention block are applied to adaptively extract and fuse multi-scale informative features. A hard example mining algorithm was also added to select important training examples to address sample imbalance. A large-scale region in western China with sufficient geological elements was set as the research area. The proposed model improved the two critical metrics by more than 2% in the experiment section. As far as we know, this research work is the first time DL features and multi-source remote sensing images have been utilized to simultaneously interpret geological elements of lithology, soil, surface water, and glaciers. The extensive experimental results demonstrated the superiority of DL features and our model in geological remote sensing interpretation.

SCPM-Net: An anchor-free 3D lung nodule detection network using sphere representation and center points matching.

Automatic and accurate lung nodule detection from 3D Computed Tomography (CT) scans plays a vital role in efficient lung cancer screening. Despite the state-of-the-art performance obtained by recent anchor-based detectors using Convolutional Neural Networks (CNNs) for this task, they require predetermined anchor parameters such as the size, number, and aspect ratio of anchors, and have limited robustness when dealing with lung nodules with a massive variety of sizes. To overcome these problems, we propose a 3D sphere representation-based center-points matching detection network (SCPM-Net) that is anchor-free and automatically predicts the position, radius, and offset of nodules without manual design of nodule/anchor parameters. The SCPM-Net consists of two novel components: sphere representation and center points matching. First, to match the nodule annotation in clinical practice, we replace the commonly used bounding box with our proposed bounding sphere to represent nodules with the centroid, radius, and local offset in 3D space. A compatible sphere-based intersection over-union loss function is introduced to train the lung nodule detection network stably and efficiently. Second, we empower the network anchor-free by designing a positive center-points selection and matching (CPM) process, which naturally discards pre-determined anchor boxes. An online hard example mining and re-focal loss subsequently enable the CPM process to be more robust, resulting in more accurate point assignment and mitigation of class imbalance. In addition, to better capture spatial information and 3D context for the detection, we propose to fuse multi-level spatial coordinate maps with the feature extractor and combine them with 3D squeeze-and-excitation attention modules. Experimental results on the LUNA16 dataset showed that our proposed SCPM-Net framework achieves superior performance compared with existing anchor-based and anchor-free methods for lung nodule detection with the average sensitivity at 7 predefined FPs/scan of 89.2%. Moreover, our sphere representation is verified to achieve higher detection accuracy than the traditional bounding box representation of lung nodules. Code is available at:https://github.com/HiLab-git/SCPM-Net.

Beet seedling and weed recognition based on convolutional neural network and multi-modality images

Difficulties in the recognition of beet seedlings and weeds can arise from a complex background in the natural environment and a lack of light at night. In the current study, a novel depth fusion algorithm was proposed based on visible and near-infrared imagery. In particular, visible (RGB) and near-infrared images were superimposed at the pixel-level via a depth fusion algorithm and were subsequently fused into three-channel multi-modality images in order to characterize the edge details of beets and weeds. Moreover, an improved region-based fully convolutional network (R-FCN) model was applied in order to overcome the geometric modeling restriction of traditional convolutional kernels. More specifically, for the convolutional feature extraction layers, deformable convolution was adopted to replace the traditional convolutional kernel, allowing for the entire network to extract more precise features. In addition, online hard example mining was introduced to excavate the hard negative samples in the detection process for the retraining of misidentified samples. A total of four models were established via the aforementioned improved methods. Results demonstrate that the average precision of the improved optimal model for beets and weeds were 84.8% and 93.2%, respectively, while the mean average precision was improved to 89.0%. Compared with the classical R-FCN model, the performance of the optimal model was not only greatly improved, but the parameters were also not significantly expanded. Our study can provide a theoretical basis for the subsequent development of intelligent weed control robots under weak light conditions.