INRIA Dataset Research Articles

Research on Deep Learning Detection Model for Pedestrian Objects in Complex Scenes Based on Improved YOLOv7

Objective: Pedestrian detection is very important for the environment perception and safety action of intelligent robots and autonomous driving, and is the key to ensuring the safe action of intelligent robots and auto assisted driving. Methods: In response to the characteristics of pedestrian objects occupying a small image area, diverse poses, complex scenes and severe occlusion, this paper proposes an improved pedestrian object detection method based on the YOLOv7 model, which adopts the Convolutional Block Attention Module (CBAM) attention mechanism and Deformable ConvNets v2 (DCNv2) in the two Efficient Layer Aggregation Network (ELAN) modules of the backbone feature extraction network. In addition, the detection head is replaced with a Dynamic Head (DyHead) detector head with an attention mechanism; unnecessary background information around the pedestrian object is also effectively excluded, making the model learn more concentrated feature representations. Results: Compared with the original model, the log-average miss rate of the improved YOLOv7 model is significantly reduced in both the Citypersons dataset and the INRIA dataset. Conclusions: The improved YOLOv7 model proposed in this paper achieved good performance improvement in different pedestrian detection problems. The research in this paper has important reference significance for pedestrian detection in complex scenes such as small, occluded and overlapping objects.

U-Net Ensemble for Enhanced Semantic Segmentation in Remote Sensing Imagery

Semantic segmentation of remote sensing imagery stands as a fundamental task within the domains of both remote sensing and computer vision. Its objective is to generate a comprehensive pixel-wise segmentation map of an image, assigning a specific label to each pixel. This facilitates in-depth analysis and comprehension of the Earth’s surface. In this paper, we propose an approach for enhancing semantic segmentation performance by employing an ensemble of U-Net models with three different backbone networks: Multi-Axis Vision Transformer, ConvFormer, and EfficientNet. The final segmentation maps are generated through a geometric mean ensemble method, leveraging the diverse representations learned by each backbone network. The effectiveness of the base U-Net models and the proposed ensemble is evaluated on multiple datasets commonly used for semantic segmentation tasks in remote sensing imagery, including LandCover.ai, LoveDA, INRIA, UAVid, and ISPRS Potsdam datasets. Our experimental results demonstrate that the proposed approach achieves state-of-the-art performance, showcasing its effectiveness and robustness in accurately capturing the semantic information embedded within remote sensing images.

A novel low-resource consumption and high-speed hardware implementation of HOG feature extraction on FPGA for human detection

In today’s increasingly complex traffic environment, pedestrian detection has become increasingly important. The Histogram of Oriented Gradients (HOG) algorithm has been proven to be highly efficient in pedestrian detection. This paper proposes a low-resource consumption, high-speed hardware implementation for HOG algorithm. In the case of a slight sacrifice in accuracy, it increases computational speed and reduces resource consumption. Experimental results demonstrate that the implementation achieves a speed of 0.933 pixels per clock cycle and consumes 4117 look-up tables and 4.5 Kbits of block RAMs while its accuracy decreases by 1.2% on the INRIA dataset and by 0.11% on the MIT dataset.

Human Detection in Clear and Hazy Weather Based on Transfer Learning With Improved INRIA Dataset Annotation

Human Detection in Clear and Hazy Weather Based on Transfer Learning With Improved INRIA Dataset Annotation

Effect of Positive-Negative Image Ratio on the Performance of Pedestrian Detection Model

Pedestrian detection holds significant importance in computer vision, finding applications in video surveillance, human-computer interaction, and autonomous vehicles. Surprisingly, there is a scarcity of research addressing the optimal ratio of positive to negative images for training detection models. This study endeavors to fill this research gap by exploring various detection models and determining the ideal ratio. Two distinct scenarios are investigated, each characterized by an equal total image count and an equivalent number of positive images sourced from CVC-14 night/visible, night/FIR, and INRIA databases. The study leverages the Histogram of Oriented Gradient, utilizing both Support Vector Machines and Medium Neural Networks to construct the detection models. Notably, the experiments reveal that the accuracy of the models remains relatively stable, even with an increase in the ratio of negative images. However, a noteworthy inverse relationship between sensitivity and specificity emerges as the ratio escalates. The findings, guided by the Youden Index, pinpoint the optimal training ratio for pedestrian detection models, falling within the range of 1:0.5 to 1:2In the CVC-14 nighttime database, the Youden index reached 100% when the model was trained with a 1:0.5 ratio using SVM, and a total of 4500 images were employed in the training process. On the other hand, in the INRIA dataset, the Youden index exhibited its highest value at 98.50%. This occurred when both SVM and a Medium neural network were utilized to train the model with a ratio of 1:2, utilizing a total of 3000 images for the training phase. It's worth highlighting that the processing time for SVM models lags behind that of Medium Neural Networks. This disparity arises from the heightened computational complexity inherent to medium-sized neural networks, making them computationally demanding compared to SVMs. This study contributes valuable insights into the nuanced relationship between image ratios and the performance of pedestrian detection models.

A cross-stage features fusion network for building extraction from remote sensing images

ABSTRACT The deep learning-based building extraction methods produce different feature maps at different stages of the network, which contain different information features. The detailed information of the feature maps decreases along the depth of the network, and insufficiently detailed information results in limited accuracy. However, existing methods are incapable of making full use of low-level feature maps with rich details. To overcome these shortcomings, we proposed a Cross-stage Features Fusion Network (CFF-Net) for building extraction from remote sensing images. In the CFF-Net, we innovatively proposed a Cross-stage Features Fusion (CFF) module that fuses different features generated at different stages. And we used the attention mechanism to make the network more focused on important information at different scales. To further improve the accuracy of building extraction, we designed the Prediction Enhancement (PE) module, where the last convolutional layer and the feature map generated in the intermediate stage are used for prediction at the same time to enhance the final result. To evaluate the effectiveness of the proposed network, we conduct quantitative and qualitative experiments on the two publicly available datasets, i.e. the Inria dataset and the WHU datasets. CFF-Net outperformed other state-of-the-art algorithms on the two datasets in IoU and F1 metrics. The efficiency analysis reveals that the proposed CFF-Net achieves a great balance between building extraction performance and complexity/efficiency, with faster convergence and higher robustness.

Recurrent Residual Deformable Conv Unit and Multi-Head with Channel Self-Attention Based on U-Net for Building Extraction from Remote Sensing Images

Considering the challenges associated with accurately identifying building shape features and distinguishing between building and non-building features during the extraction of buildings from remote sensing images using deep learning, we propose a novel method for building extraction based on U-Net, incorporating a recurrent residual deformable convolution unit (RDCU) module and augmented multi-head self-attention (AMSA). By replacing conventional convolution modules with an RDCU, which adopts a deformable convolutional neural network within a residual network structure, the proposed method enhances the module’s capacity to learn intricate details such as building shapes. Furthermore, AMSA is introduced into the skip connection function to enhance feature expression and positions through content–position enhancement operations and content–content enhancement operations. Moreover, AMSA integrates an additional fusion channel attention mechanism to aid in identifying cross-channel feature expression Intersection over Union (IoU) score differences. For the Massachusetts dataset, the proposed method achieves an Intersection over Union (IoU) score of 89.99%, PA (Pixel Accuracy) score of 93.62%, and Recall score of 89.22%. For the WHU Satellite dataset I, the proposed method achieves an IoU score of 86.47%, PA score of 92.45%, and Recall score of 91.62%, For the INRIA dataset, the proposed method achieves an IoU score of 80.47%, PA score of 90.15%, and Recall score of 85.42%.

Research on HOG-SVM pedestrian detection method based on FPGA

At present, pedestrian detection technology has become a research hotspot in the field of image processing and computer vision, and has developed to open up a wide range of application prospects. This paper gives a comprehensive introduction to the algorithm and FPGA implementation of object detection algorithm including HOG and SVM. In the field of pedestrian detection, HOG-SVM pedestrian detection method based on FPGA has unique advantages compared with other various object detection methods. In the current research, this implementation method usually has a detection accuracy rate of more than 95% and a detection speed of more than 30 frames/s for the INRIA data sets, which perfectly meets the pedestrian detection requirement. FPGA with its parallel structure and a large amount of programmable logic sources greatly improve the HOG-SVM to perform real-time pedestrian detection. With the development of hardware acceleration and detection algorithms, more and more pedestrian detection methods with better performance will appear in the future.

An H-GrabCut Image Segmentation Algorithm for Indoor Pedestrian Background Removal.

In the context of predicting pedestrian trajectories for indoor mobile robots, it is crucial to accurately measure the distance between indoor pedestrians and robots. This study aims to address this requirement by extracting pedestrians as regions of interest and mitigating issues related to inaccurate depth camera distance measurements and illumination conditions. To tackle these challenges, we focus on an improved version of the H-GrabCut image segmentation algorithm, which involves four steps for segmenting indoor pedestrians. Firstly, we leverage the YOLO-V5 object recognition algorithm to construct detection nodes. Next, we propose an enhanced BIL-MSRCR algorithm to enhance the edge details of pedestrians. Finally, we optimize the clustering features of the GrabCut algorithm by incorporating two-dimensional entropy, UV component distance, and LBP texture feature values. The experimental results demonstrate that our algorithm achieves a segmentation accuracy of 97.13% in both the INRIA dataset and real-world tests, outperforming alternative methods in terms of sensitivity, missegmentation rate, and intersection-over-union metrics. These experiments confirm the feasibility and practicality of our approach. The aforementioned findings will be utilized in the preliminary processing of indoor mobile robot pedestrian trajectory prediction and enable path planning based on the predicted results.

ARE-Net: An Improved Interactive Model for Accurate Building Extraction in High-Resolution Remote Sensing Imagery

Accurate building extraction for high-resolution remote sensing images is critical for topographic mapping, urban planning, and many other applications. Its main task is to label each pixel point as a building or non-building. Although deep-learning-based algorithms have significantly enhanced the accuracy of building extraction, fully automated methods for building extraction are limited by the requirement for a large number of annotated samples, resulting in a limited generalization ability, easy misclassification in complex remote sensing images, and higher costs due to the need for a large number of annotated samples. To address these challenges, this paper proposes an improved interactive building extraction model, ARE-Net, which adopts a deep interactive segmentation approach. In this paper, we present several key contributions. Firstly, an adaptive-radius encoding (ARE) module was designed to optimize the interaction features of clicks based on the varying shapes and distributions of buildings to provide maximum a priori information for building extraction. Secondly, a two-stage training strategy was proposed to enhance the convergence speed and efficiency of the segmentation process. Finally, some comprehensive experiments using two models of different sizes (HRNet18s+OCR and HRNet32+OCR) were conducted on the Inria and WHU building datasets. The results showed significant improvements over the current state-of-the-art method in terms of NoC90. The proposed method achieved performance enhancements of 7.98% and 13.03% with HRNet18s+OCR and 7.34% and 15.49% with HRNet32+OCR on the WHU and Inria datasets, respectively. Furthermore, the experiments demonstrated that the proposed ARE-Net method significantly reduced the annotation costs while improving the convergence speed and generalization performance.

Exploring Relationships between Boltzmann Entropy of Images and Building Classification Accuracy in Land Cover Mapping.

Remote sensing images are important data sources for land cover mapping. As one of the most important artificial features in remote sensing images, buildings play a critical role in many applications, such as population estimation and urban planning. Classifying buildings quickly and accurately ensures the reliability of the above applications. It is known that the classification accuracy of buildings (usually indicated by a comprehensive index called F1) is greatly affected by image quality. However, how image quality affects building classification accuracy is still unclear. In this study, Boltzmann entropy (an index considering both compositional and configurational information, simply called BE) is employed to describe image quality, and the potential relationships between BE and F1 are explored based on images from two open-source building datasets (i.e., the WHU and Inria datasets) in three cities (i.e., Christchurch, Chicago and Austin). Experimental results show that (1) F1 fluctuates greatly in images where building proportions are small (especially in images with building proportions smaller than 1%) and (2) BE has a negative relationship with F1 (i.e., when BE becomes larger, F1 tends to become smaller). The negative relationships are confirmed using Spearman correlation coefficients (SCCs) and various confidence intervals via bootstrapping (i.e., a nonparametric statistical method). Such discoveries are helpful in deepening our understanding of how image quality affects building classification accuracy.

TCNN Architecture for Partial Occlusion Handling in Pedestrian Classification

Pedestrian classification is of increased interest to autonomous transportation systems due to the development of deep convolutional neural networks. Despite recent progress on pedestrian classification, it is still challenging to identify individuals who are partially occluded because of the diversity of the occluded parts, variation in pose, and appearance. This causes a significant performance reduction when pedestrians are covered by other objects, and feature information is lost due to the occluded parts. To solve this problem, we propose two network architectures using tree structure convolutional neural networks (T-CNN). They use the structural representation of multi-branch deep convolutional features, with the advantages of its end-to-end learning process. The high-level tree structure CNN (HT-CNN) architecture aims to concatenate the output of the classification layer from multi-segmented patches of pedestrians to handle partially occluded problems. The low-level tree structure CNN (LT-CNN) concatenates the discriminative features from each multi-segmented patch and global features. Our T-CNN architecture with a high-level tree structure performed with 94.64% accuracy on the INRIA dataset without occlusions, and with 70.78% accuracy on the Prince of Songkla University (PSU) dataset with occlusions, outperforming a baseline CNN architecture. This indicates that our proposed architecture can be used in a real-world environment to classify the occluded part of pedestrians with the visual information of multi-segmented patches using tree-structured multi-branched CNN.

Improving building rooftop segmentation accuracy through the optimization of UNet basic elements and image foreground-background balance

Building rooftop segmentation using deep learning techniques is a popular yet challenging area of research in computer vision and remote sensing image processing. While recent studies have developed various deep learning models, there has been less focus on investigating the basic elements of network architecture and foreground-background balance in sampled images. To address this research gap, this study proposes optimizing the UNet basic elements and image foreground-background balance to improve building rooftop segmentation accuracy. The Inria dataset is used for model training and accuracy evaluation. The results show that the impact of network backbone on segmentation accuracy depends on its ability to extract lower-level features and whether it is pretrained by large amounts of data. This study also finds that residual structures of network backbone can negatively affect performance regardless of the network depth. Among the decoding network upsampling strategies, the proposed super-resolution method achieves the best segmentation results, but does not significantly differ from the commonly used bilinear interpolation and deconvolution methods. Additionally, this study proposes a hybrid loss function that considers area, arithmetic, shape, spectral, and texture discrepancy, which further improves the segmentation performance. Finally, this study highlights the significant impact of image foreground-background balance on building rooftop segmentation and proposes an ensemble model to mitigate image foreground-background imbalance and improve segmentation performance.

PolyBuilding: Polygon transformer for building extraction

We present PolyBuilding, a polygon Transformer for building extraction. PolyBuilding direct predicts vector representation of buildings from remote sensing images. It builds upon an encoder–decoder transformer architecture and simultaneously predicts the bounding boxes and polygons for the building instances. Given a set of polygon queries, the model learns the relations among them and encodes context information from the image to predict the final set of building polygons with a fixed vertex number. Considering that predicting a fixed number of vertices would cause vertex redundancy and reduce polygon regularity, we design a corner classification head to distinguish the building corners. By taking advantage of the corner classification scores, a polygon refinement scheme is designed to remove the redundant vertices and produce the final polygons with regular contours and low complexity. In addition, although the PolyBuilding model is fully end-to-end trainable, we propose a two-phase training strategy to decompose the coordinate regression and corner classification into two stages to alleviate the difficulty of multi-task learning. Comprehensive experiments are conducted on the CrowdAI dataset and Inria dataset. PolyBuilding achieves a new state-of-the-art in terms of pixel-level coverage, instance-level detection performance, and geometry-level properties. Quantitative and qualitative results verify the superiority and effectiveness of our model.

HiSup: Accurate polygonal mapping of buildings in satellite imagery with hierarchical supervision

This paper studies the problem of the polygonal mapping of buildings by tackling the issue of mask reversibility, which leads to a notable performance gap between the predicted masks and polygons from the learning-based methods. We addressed such an issue by exploiting the hierarchical supervision (of bottom-level vertices, mid-level line segments, and high-level regional masks) and proposed a novel interaction mechanism of feature embedding sourced from different levels of supervision signals to obtain reversible building masks for polygonal mapping of buildings. As a result, we show that the learned reversible building masks take all the merits of the advances of deep convolutional neural networks for high-performing polygonal mapping of buildings. In the experiments, we evaluated our method on four public benchmarks, including the AICrowd, Open Cities, Shanghai, and Inria datasets. On the AICrowd, Open Cities, and Shanghai datasets, our proposed method obtains unanimous improvements on the metrics of AP, APboundary and PoLiS by large margins. For the Inria dataset, our proposed method also obtains very competitive results on the metrics of IoU and Accuracy. The models and source code are available at https://github.com/SarahwXU/HiSup.

MAE-BG: dual-stream boundary optimization for remote sensing image semantic segmentation

Deep learning has achieved remarkable performance in semantically segmenting remotely sensed images. However, the high-frequency detail loss caused by continuous convolution and pooling operations and the uncertainty introduced when annotating low-contrast objects with weak boundaries induce blurred object boundaries. Therefore, a dual-stream network MAE-BG, consisting of an edge detection (ED) branch and a smooth branch with boundary guidance (BG), is proposed. The ED branch is designed to enhance the weak edges that need to be preserved, simultaneously suppressing false responses caused by local texture. This mechanism is achieved by introducing improved multiple-attention edge detection blocks (MAE). Furthermore, two specific ED branches with MAE are designed to combine with typical deep convolutional (DC) and Codec infrastructures and result in two configurations of MAE-A and MAE-B. Meanwhile, multiscale edge information extracted by MAE networks is fed into the backbone networks to complement the detail loss caused by convolution and pooling operations. This results in smooth networks with BG. After that, the segmentation results with improved boundaries are obtained by stacking the output of the ED and smooth branches. The proposed algorithms were evaluated on the ISPRS Potsdam and Inria Aerial Image Labelling datasets. Comprehensive experiments show that the proposed method can precisely locate object boundaries and improve segmentation performance. The MAE-A branch leads to an overall accuracy (OA) of 89.16%, a mean intersection over union (MIOU) of 80.25% for Potsdam, and an OA of 96.61% and MIOU of 86.63% for Inria. Compared with the results without the proposed edge optimization blocks, the OAs from the Potsdam and Inria datasets increase by 5.49% and 7.64%, respectively.

FSAU-Net: a network for extracting buildings from remote sensing imagery using feature self-attention

ABSTRACT Convolutional neural networks (CNNs) extract semantic features from images by stacking convolutional operators, which easily causes semantic information loss and leads to hollow and edge inaccuracies in building extraction. Therefore, a features self-attention U-block network (FSAU-Net) is proposed. The network focuses on the target feature self-attention in the coding stage, and features self-attention (FSA) distinguishes buildings from nonbuilding by weighting the extracted features themselves; we introduce spatial attention (SA) in the decoder stage to focus on the spatial locations of features, and SA generates spatial location features through the spatial relationship among the features to highlight the building information area. A jump connection is used to fuse the shallow features generated in the decoder stage with the deep features generated in the encoder stage to reduce the building information loss. We validate the superiority of the method FSAU-Net on the WHU and Inria datasets with 0.3 m resolution and Massachusetts with 1.0 m resolution, experimentally showing IoU of 91.73%, 80.73% and 78.46% and precision of 93.60%, 90.71% and 86.37%, respectively. In addition, we also set up ablation experiments by adding an FSA module, Squeeze-and-Excitation (SE) module and Efficient Channel Attention (ECA) module to UNet and ResNet101, where UNet+FSA improves the IoU values by 3.15%, 2.72% and 1.77% compared to UNet, UNet+SE and UNet+ECA, respectively, and ResNet101+FSA improves the IoU values by 2.06%, 1.17% and 0.9% compared to ResNet101, ResNet101+SE and ResNet101+ECA, respectively, demonstrating the superiority of our proposed FSA module. FSAU-Net improves the IoU values by 3.18%, 2.75% and 1.80% compared to those of UNet, UNet+SE and UNet+ECA, respectively. FSAU-Net has 2.11%, 1.22%, and 0.95% IoU improvements over the IoU values of ResNet101, ResNet101+SE and ResNet101+ECA, respectively, demonstrating the superiority of our proposed FSAU-Net model. The TensorFlow implementation is available at https://github.com/HMH12456/FSAU-Net-master.git.

SSAD: Single-Shot Multi-Scale Attentive Detector for Autonomous Driving

A novel Single-shot Multi-scale detection network with feature fusion and multi-scale attention mechanism is proposed for autonomous driving. The proposed model is referred to as a Single-shot Multi-scale Attentive Detector (SSAD) and it would build feature relations of the feature map in the spatial space. The proposed network highlights pedestrian and vehicle regions on the extracted feature map and also suppresses irrelevant regions, from the global relation information and thereby providing reliable guidance for autonomous driving, while detecting smaller and occluded targets. SSAD design is simple, accurate and computationally efficient. Evaluation results of the proposed network on multiple datasets have achieved considerable and promising results. Experimental results show that the SSAD network when tested on Pascal Voc-2007, INRIA, Caltech, and City Persons datasets, outperforms many state-of-the-art (SOTA) detectors.

Multimodal pedestrian detection using metaheuristics with deep convolutional neural network in crowded scenes

Pedestrian detection (PD) is a vital computer vision (CV) problem that is highly employed in several real-time applications, namely autonomous driving methods, robotics, and security observing methods. Simulated by deep learning (DL) approaches to the recognition of generic objects, several investigation mechanisms have attained maximum recognition accuracy for acceptable scale and non-blocked pedestrians. However, the detection efficiency needed to be improved for complex cases like rare pose samples, crowd scenes, and cases with worse visibility due to daytime or weather. Therefore, this study develops a multimodal pedestrian detection system in crowded scenes using metaheuristics and a deep convolutional neural network (MMPD-MDCNN) technique. The MMPD-MDCNN technique’s goal is to identify pedestrians in crowd scenes using different deep-learning models effectively. The proposed MMPD-MDCNN technique integrates three deep learning models: the residual network (ResNet-50), Inception v3, and the capsule network (CapsNet). In addition, the Harris Hawks Optimization (HHO) algorithm is applied for optimal hyperparameter tuning of the deep learning models. For pedestrian detection, the MMPD-MDCNN technique uses the long short-term memory (LSTM) model, and its hyperparameters can be adjusted by the shark smell optimization (SSO) algorithm. To demonstrate the superior performance of the MMPD-MDCNN approach, A comprehensive set of simulations on the INRIA and UCSD datasets was performed to illustrate the superior performance of the MMPD-MDCNN approach. The experimental results suggest that the MMPD-MDCNN model performs well on both datasets.

A Multi-Scale Edge Constraint Network for the Fine Extraction of Buildings from Remote Sensing Images

Building extraction based on remote sensing images has been widely used in many industries. However, state-of-the-art methods produce an incomplete segmentation of buildings owing to unstable multi-scale context aggregation and a lack of consideration of semantic boundaries, ultimately resulting in large uncertainties in predictions at building boundaries. In this study, efficient fine building extraction methods were explored, which demonstrated that the rational use of edge features can significantly improve building recognition performance. Herein, a fine building extraction network based on a multi-scale edge constraint (MEC-Net) was proposed, which integrates the multi-scale feature fusion advantages of UNet++ and fuses edge features with other learnable multi-scale features to achieve the effect of prior constraints. Attention was paid to the alleviation of noise interference in the edge features. At the data level, according to the improvement of copy-paste according to the characteristics of remote sensing imaging, a data augmentation method for buildings (build-building) was proposed, which increased the number and diversity of positive samples by simulating the construction of buildings to increase the generalization of MEC-Net. MEC-Net achieved 91.13%, 81.05% and 74.13% IoU on the WHU, Massachusetts and Inria datasets, and it has a good inference efficiency. The experimental results show that MEC-Net outperforms the state-of-the-art methods, demonstrating its superiority. MEC-Net improves the accuracy of building boundaries by rationally using previous edge features.