Region Proposal Network Research Articles

Determining the proper number of proposals for individual images

AbstractThe region proposal network is indispensable to two‐stage object detection methods. It generates a fixed number of proposals that are to be classified and regressed by detection heads to produce detection boxes. However, the fixed number of proposals may be too large when an image contains only a few objects but too small when it contains much more objects. Considering this, the authors explored determining a proper number of proposals according to the number of objects in an image to reduce the computational cost while improving the detection accuracy. Since the number of ground truth objects is unknown at the inference stage, the authors designed a simple but effective module to predict the number of foreground regions, which will be substituted for the number of objects for determining the proposal number. Experimental results of various two‐stage detection methods on different datasets, including MS‐COCO, PASCAL VOC, and CrowdHuman showed that equipping the designed module increased the detection accuracy while decreasing the FLOPs of the detection head. For example, experimental results on the PASCAL VOC dataset showed that applying the designed module to Libra R‐CNN and Grid R‐CNN increased over 1.5 AP50 while decreasing the FLOPs of detection heads from 28.6 G to nearly 9.0 G.

Center TextSpotter: A Novel Text Spotter for Autonomous Unmanned Vehicles

Automatic text spotting can help unmanned vehicles read human texts, thus improving the safety and reliability of autonomous driving. Some existing text spotting models use inefficient region proposal networks or modules based on the recurrent neural network. Region proposal networks produce redundant anchors. Recurrent neural networks cannot perform parallel operations well. The existence of these inefficient modules prevents these models from applying in autonomous unmanned vehicles. In this paper, we propose an end-to-end two-stage text spotting model named Center TextSpotter, which is a convolution model that does not involve region proposal networks and recurrent neural network. Moreover, we develop a weakly supervised training method and a feature fusion module. The weakly supervised training method adjusts the loss by weighting the predicted labels to improve the performance of text recognition. The feature fusion module fuses the features of the proposal with the features of the proposal-context to enhance the overall performance. Our model follows the modular design principle. So it can be easily extended and modified. This paper presents an extension scheme based on graph neural network. By adding the graph neural network before the fully connected layer, the model learns better features. Experiments demonstrate that Center TextSpotter can complete the task of text spotting for autonomous unmanned vehicles commendably.

Tunnel Lining Crack Recognition Algorithm Integrating SK Attention and Cascade Neural Network

Cracks are one of the main types of tunnel-lining defects. At present, there are no particularly good methods for identifying tunnel-lining cracks. The methods that are used are associated with problems such as poor robustness, low detection efficiency, and inconsistency in defect identification. Vision-based crack identification algorithms that use images of tunnel-lining cracks are also affected by problems such as weak features, irregular shapes, random development, and small proportions. In light of this, we propose in this paper a neural network that integrates U-net and FPN (feature pyramid network) pyramid structure characteristics. Because of its ability to output deep feature information, an SK (selective kernel) attention mechanism was integrated to increase the weight of effective feature regions and improve the detection accuracy of feature maps. An RPN (region proposal network) regional detection network was used for multi-scale regional screening, and non-maximum suppression was used to eliminate repeated anchor frames between different regions. The continuity of cracks was determined using adaptive regional expansion. The region segmentation network was also combined with multi-scale feature maps. Finally, the segmentation results for regions of fine features could be mapped to the whole image to identify cracks, thereby solving the problem that arises when textural features of cracks are weak and cannot be accurately identified. For testing purposes, photos of tunnel-lining crack defects were used. The algorithm described in this paper was able to combine deep and shallow features to identify more abundant crack features, and the recognition accuracy of the crack classification network reached 98.51%. With respect to crack segmentation, the algorithm’s segmentation accuracy reached 94.55%, and its single processing time reached 60.76 ms, indicating more accurate and efficient segmentation performance, compared with FCNs (fully convolutional networks) and classical U-net networks.

Research on Vision of Intelligent Car Based on Broad Learning System.

The broad learning system (BLS) of intelligent vehicle in different target environments is studied in this article. First, this article provides with the target recognition image data to be trained and detected through the automated guided vehicle (AGV) mobile platform, which can grab the recognition image of different angles and backgrounds. In order to avoid the data generalization phenomenon, the dataset can be expanded by the data normalization and data enhancement. Second, the data are input into the shared convolution layer to extract the feature image and maintain the image. The parameters of image height, width, and channel number are invariable, and the new feature image is obtained by further extraction. Furthermore, the region proposal network (RPN) prefiltering algorithm based on hierarchical clustering is used to filter the objects in the candidate box to determine the region image corresponding to the feature image. Then, the feature images of different sizes input into region of interest (ROI) pooling are used to keep the size of the image in the ROI consistent. Finally, the normalized image is input into the classifier module to obtain the category of the target recognition image to be detected. Through the simulation experiments of different groups, it can be seen that the target recognition system proposed in this design can not only accurately detect the objects but also stably recognize the objects in different environments. The target recognition accuracy for the optimized system is about 95%.

A Tiny Object Detection Approach for Maize Cleaning Operations.

Real-time and accurate awareness of the grain situation proves beneficial for making targeted and dynamic adjustments to cleaning parameters and strategies, leading to efficient and effective removal of impurities with minimal losses. In this study, harvested maize was employed as the raw material, and a specialized object detection network focused on impurity-containing maize images was developed to determine the types and distribution of impurities during the cleaning operations. On the basis of the classic contribution Faster Region Convolutional Neural Network, EfficientNetB7 was introduced as the backbone of the feature learning network and a cross-stage feature integration mechanism was embedded to obtain the global features that contained multi-scale mappings. The spatial information and semantic descriptions of feature matrices from different hierarchies could be fused through continuous convolution and upsampling operations. At the same time, taking into account the geometric properties of the objects to be detected and combining the images' resolution, the adaptive region proposal network (ARPN) was designed and utilized to generate candidate boxes with appropriate sizes for the detectors, which was beneficial to the capture and localization of tiny objects. The effectiveness of the proposed tiny object detection model and each improved component were validated through ablation experiments on the constructed RGB impurity-containing image datasets.

FRDet: Few‐shot object detection via feature reconstruction

AbstractState‐of‐the‐art object detection models rely on large‐scale datasets for training to achieve good precision. Without sufficient samples, the model can suffer from severe overfitting. Current explorations in few‐shot object detection are mainly divided into meta‐learning‐based methods and fine‐tuning‐based methods. However, existing models do not focus on how feature maps should be processed to present more accurate regions of interest (RoIs), leading to many non‐supporting RoIs. These non‐supporting RoIs can increase the burden of subsequent classification and even lead to misclassification. Additionally, catastrophic forgetting is inevitable in both few‐shot object detection models. Many models classify directly in low‐dimensional spaces due to insufficient resources, but this transformation of the data space can confuse some categories and lead to misclassification. To address these problems, the Feature Reconstruction Detector (FRDet) is proposed, a simple yet effective fine‐tune‐based approach for few‐shot object detection. FRDet includes a region proposal network (RPN) based on channel attention and space attention called Multi‐Attention RPN (MARPN) and a head based on feature reconstruction called Feature Reconstruction Head (FRHead). MARPN utilizes channel attention to suppress non‐supporting classes and spatial attention to enhance support classes based on Attention RPN, resulting in fewer but more accurate RoIs. Meanwhile, FRHead utilizes support features to reconstruct query RoI features through a closed‐form solution, allowing for a comprehensive and fine‐grained comparison. The model was validated on the PASCAL VOC, MS COCO, FSOD, and CUB200 datasets and achieved better results.

Infrared Dim and Small Target Detection Based on Background Prediction

Infrared dim and small target detection is a key technology for various detection tasks. However, due to the lack of shape, texture, and other information, it is a challenging task to detect dim and small targets. Recently, since many traditional algorithms ignore the global information of infrared images, they generate some false alarms in complicated environments. To address this problem, in this paper, a coarse-to-fine deep learning-based method was proposed to detect dim and small targets. Firstly, a coarse-to-fine detection framework integrating deep learning and background prediction was applied for detecting targets. The framework contains a coarse detection module and a fine detection module. In the coarse detection stage, Region Proposal Network (RPN) is employed to generate masks in target candidate regions. Then, to further optimize the result, inpainting is utilized to predict the background using the global semantics of images. In this paper, an inpainting algorithm with a mask-aware dynamic filtering module was incorporated into the fine detection stage to estimate the background of the candidate targets. Finally, compared with existing algorithms, the experimental results indicate that the proposed framework has effective detection capability and robustness for complex surroundings.

Region-based Convolutional Neural Network (R-CNN) architecture for auto-cropping of pancreatic computed tomography

ABSTRACT Automatic pancreas detection and cropping with high precision from medical images is an important yet challenging problem for medical image analysis and Computer-Aided Diagnosis (CAD). Factors relating to the limited availability of image data and segmentation methodology hinder this task. High variability in the location of the pancreas,which occupies a very small area of the pancreatic Computed Tomography (CT) scans,and the anatomy of organs also add to the list of issues. These challenges necessitate an urgent need for the development of localization and auto-cropping methods of the region of interest (ROI). This paper presents the results obtained by the implementation of Region-based Convolutional Neural Network (RCNN)-Crop inspired by the Region Proposal Network (RPN) and Feature Pyramid Network (FPN) to localize the pancreas by building bounding boxes and auto-crop the ROI obtained from various other organs in the pancreatic CT scans and has a Mean Average Precision (mAP) of 28.10% for the dataset provided.

"Towards Real-time House Detection in Aerial Imagery Using Faster Region-based Convolutional Neural Network"

In the past few years, automatic building detection in aerial images has become an emerging field in computer vision. Detecting the specific types of houses will provide information in urbanization, change detection, and urban monitoring that play increasingly important roles in modern city planning and natural hazard preparedness. In this paper, we demonstrate the effectiveness of detecting various types of houses in aerial imagery using Faster Region-based Convolutional Neural Network (Faster-RCNN). After formulating the dataset and extracting bounding-box information, pre-trained ResNet50 is used to get the feature maps. The fully convolutional Region Proposal Network (RPN) first predicts the bounds and objectness score of objects (in this case house) from the feature maps. Then, the Region of Interest (RoI) pooling layer extracts interested regions to detect objects that are present in the images. To the best of our knowledge, this is the first attempt at detecting houses using Faster R-CNN that has achieved satisfactory results. This experiment opens a new path to conduct and extent the works not only for civil and environmental domain but also other applied science disciplines.

Fine-Grained Image Recognition by Means of Integrating Transformer Encoder Blocks in a Robust Single-Stage Object Detector

Fine-grained image classification remains an ongoing challenge in the computer vision field, which is particularly intended to identify objects within sub-categories. It is a difficult task since there is both minimal and substantial intra-class variance. Current methods address the issue through first locating selective regions with region proposal networks (RPNs), object localization, or part localization, followed by implementing a CNN network or SVM classifier to those selective regions. This approach, however, makes the process simple via implementing a single-stage end-to-end feature encoded with a localization method, which leads to improved feature representations of individual tokens/regions through integrating the transformer encoder blocks into the Yolov5 backbone structure. These transformer encoder blocks, with their self-attention mechanism, effectively capture global dependencies and enable the model to learn relationships between distant regions. This improves the model’s ability to understand context and capture long-range spatial relationships in an image. We also replaced the Yolov5 detection heads with three transformer heads at the output for object recognition using the discriminative and informative feature maps from transformer encoder blocks. We established the potential of the single-stage detector for the fine-grained image recognition task, achieving state-of-the-art 93.4% accuracy, as well as outperforming existing one-stage recognition models. The effectiveness of our approach is assessed using the Stanford car dataset, which includes 16,185 images of 196 different classes of vehicles with significantly identical visual appearances.

Locate Then Generate: Bridging Vision and Language with Bounding Box for Scene-Text VQA

In this paper, we propose a novel multi-modal framework for Scene Text Visual Question Answering (STVQA), which requires models to read scene text in images for question answering. Apart from text or visual objects, which could exist independently, scene text naturally links text and visual modalities together by conveying linguistic semantics while being a visual object in an image simultaneously. Different to conventional STVQA models which take the linguistic semantics and visual semantics in scene text as two separate features, in this paper, we propose a paradigm of "Locate Then Generate" (LTG), which explicitly unifies this two semantics with the spatial bounding box as a bridge connecting them. Specifically, at first, LTG locates the region in an image that may contain the answer words with an answer location module (ALM) consisting of a region proposal network and a language refinement network, both of which can transform to each other with one-to-one mapping via the scene text bounding box. Next, given the answer words selected by ALM, LTG generates a readable answer sequence with an answer generation module (AGM) based on a pre-trained language model. As a benefit of the explicit alignment of the visual and linguistic semantics, even without any scene text based pre-training tasks, LTG can boost the absolute accuracy by +6.06% and +6.92% on the TextVQA dataset and the ST-VQA dataset respectively, compared with a non-pre-training baseline. We further demonstrate that LTG effectively unifies visual and text modalities through the spatial bounding box connection, which is underappreciated in previous methods.

GLT-T: Global-Local Transformer Voting for 3D Single Object Tracking in Point Clouds

Current 3D single object tracking methods are typically based on VoteNet, a 3D region proposal network. Despite the success, using a single seed point feature as the cue for offset learning in VoteNet prevents high-quality 3D proposals from being generated. Moreover, seed points with different importance are treated equally in the voting process, aggravating this defect. To address these issues, we propose a novel global-local transformer voting scheme to provide more informative cues and guide the model pay more attention on potential seed points, promoting the generation of high-quality 3D proposals. Technically, a global-local transformer (GLT) module is employed to integrate object- and patch-aware prior into seed point features to effectively form strong feature representation for geometric positions of the seed points, thus providing more robust and accurate cues for offset learning. Subsequently, a simple yet effective training strategy is designed to train the GLT module. We develop an importance prediction branch to learn the potential importance of the seed points and treat the output weights vector as a training constraint term. By incorporating the above components together, we exhibit a superior tracking method GLT-T. Extensive experiments on challenging KITTI and NuScenes benchmarks demonstrate that GLT-T achieves state-of-the-art performance in the 3D single object tracking task. Besides, further ablation studies show the advantages of the proposed global-local transformer voting scheme over the original VoteNet. Code and models will be available at https://github.com/haooozi/GLT-T.

A Domain-Adaptive Tree-Crown Detection and Counting Method Based on Cascade Region Proposal Networks

Rapid and accurate tree-crown detection is significant to forestry management and precision forestry. In the past few decades, the development and maturity of remote sensing technology has created more convenience for tree-crown detection and planting management. However, the variability of the data source leads to significant differences between feature distributions, bringing great challenges for traditional deep-learning-based methods on cross-regional detection. Moreover, compared with other tasks, tree-crown detection has the problems of a poor abundance of objects, an overwhelming number of easy samples and the existence of a quantity of impervious background similar to the tree crown, which make it difficult for the classifier to learn discriminative features. To solve these problems, we apply domain adaptation (DA) to tree-crown detection and propose a DA cascade tree-crown detection framework with multiple region proposal networks, dubbed CAS-DA, realizing cross-regional tree-crown detection and counting from multiple-source remote sensing images. The essence of the multiple region proposal networks in CAS-DA is obtaining the multilevel features and enhancing deeper label classifiers gradually by filtering simple samples of source domain at an early stage. Then, the cascade structure is integrated with a DA object detector and the end-to-end training is realized through the proposed cascade loss function. Moreover, a filtering strategy based on the planting rules of tree crowns is designed and applied to filter wrongly detected trees by CAS-DA. We verify the effectiveness of our method in two different domain shift scenarios, including adaptation between satellite and drone images and cross-satellite adaptation. The results show that, compared to the existing DA methods, our method achieves the best average F1-score in all adaptions. It is also found that the performance between satellite and drone images is significantly worse than that between different satellite images, with average F1-scores of 68.95% and 88.83%, respectively. Nevertheless, there is an improvement of 11.88%~40.00% in the former, which is greater than 0.50%~5.02% in the latter. The above results prove that in tree-crown detection, it is more effective for the DA detector to improve the detection performance on the source domain than to diminish the domain shift alone, especially when a large domain shift exists.

Enhancing Object Detection in Self-Driving Cars Using a Hybrid Approach

Recent advancements in artificial intelligence (AI) have greatly improved the object detection capabilities of autonomous vehicles, especially using convolutional neural networks (CNNs). However, achieving high levels of accuracy and speed simultaneously in vehicular environments remains a challenge. Therefore, this paper proposes a hybrid approach that incorporates the features of two state-of-the-art object detection models: You Only Look Once (YOLO) and Faster Region CNN (Faster R-CNN). The proposed hybrid approach combines the detection and boundary box selection capabilities of YOLO with the region of interest (RoI) pooling from Faster R-CNN, resulting in improved segmentation and classification accuracy. Furthermore, we skip the Region Proposal Network (RPN) from the Faster R-CNN architecture to optimize processing time. The hybrid model is trained on a local dataset of 10,000 labeled traffic images collected during driving scenarios, further enhancing its accuracy. The results demonstrate that our proposed hybrid approach outperforms existing state-of-the-art models, providing both high accuracy and practical real-time object detection for autonomous vehicles. It is observed that the proposed hybrid model achieves a significant increase in accuracy, with improvements ranging from 5 to 7 percent compared to the standalone YOLO models. The findings of this research have practical implications for the integration of AI technologies in autonomous driving systems.

An Efficient and Accurate Cross-Domain Object Detection Method Using One-Level Feature and Domain Adaptation

To improve the accuracy of cross-domain object detection, the existing unsupervised domain adaptation (UDA) object detection methods mostly use Feature Pyramid Network (FPN), multiple Region Proposal Network (RPN), and multiple domain classifier, but these methods lead to complex network structures, slow model convergence, and low detection efficiency. To solve the above problems, this paper proposes an Efficient and Accurate Cross-domain Object Detection Method Using One-level Feature and Domain Adaptation (OFDA). This method realizes one-level feature object detection through feature fusion and divide-and-conquer technology; realizes overfitting feature suppression and unsupervised domain adaptation through domain-specific suppression and domain feature alignment technology; realizes background feature suppression through the Objectness branch, which replaces the time-consuming Region Proposal Network (RPN) structure and improves the efficiency of unsupervised adaptive detection. The paper verifies the feasibility and superiority of the proposed method by the comparative experiments and ablation experiments of multiple datasets. The proposed OFDA method not only improves the efficiency of object detection, but also ensures the accuracy of cross-domain detection.

Research on a Traffic Sign Recognition Method under Small Sample Conditions.

Traffic signs are updated quickly, and there image acquisition and labeling work requires a lot of manpower and material resources, so it is difficult to provide a large number of training samples for high-precision recognition. Aiming at this problem, a traffic sign recognition method based on FSOD (few-shot object learning) is proposed. This method adjusts the backbone network of the original model and introduces dropout, which improves the detection accuracy and reduces the risk of overfitting. Secondly, an RPN (region proposal network) with improved attention mechanism is proposed to generate more accurate target candidate boxes by selectively enhancing some features. Finally, the FPN (feature pyramid network) is introduced for multi-scale feature extraction, and the feature map with higher semantic information but lower resolution is merged with the feature map with higher resolution but weaker semantic information, which further improves the detection accuracy. Compared with the baseline model, the improved algorithm improves the 5-way 3-shot and 5-way 5-shot tasks by 4.27% and 1.64%, respectively. We apply the model structure to the PASCAL VOC dataset. The results show that this method is superior to some current few-shot object detection algorithms.

Cork classification based on multi-scale faster-RCNN with machine vision

In traditional industrial production, the classification of cork mainly adopts manual operation. Due to round cork blocks with complex textures, manual sorting is time-consuming and often leads to inefficiency. It will lead to inefficiency and a high error rate. Therefore, an automatic cork sorting system is urgently required to replace manual operation. In this paper, an advanced deep-learning method based on machine vision is adopted to detect the surface of corks and classify them automatically. To robustly detect the texture of cork profile, a Multiple Scale Faster R-CNN based on the upper and lower layers (UMS-FRCNN) approach is proposed. The network consists of a region proposal component and the region-of-interest (RoI) detection component. There are two main contributions in our proposed network, which play a significant role in achieving state-of-the-art performance in cork classification. Firstly, the multi-scale information is grouped in two aspects: region suggestion and RoI detection, to detect on small cork texture regions. Secondly, the convolution Region Proposal Network (RPN) is used to generate the candidate regions of the UMS-FRCNN model to improve the classification accuracy. The experimental results show that our method not only can accurately classify the corks but also has better performance than the Faster RCNN. Compared with other methods, our method has the advantages of high precision, less time-consuming, and no dependence labeling. Our work has been successfully utilized in a cork production plant.

An active foveated gaze prediction algorithm based on a Bayesian ideal observer

Predicting human eye movements is a crucial task for understanding human behavior and has numerous applications in machine vision. Most current models for predicting eye movements are data-driven and require large datasets of recorded eye movements, which can be expensive and time-consuming to collect. In this paper, we present a novel theory-based model for predicting eye movements in a foveated visual system that maximizes information gain at each fixation. Our model uses a region-proposal network and eccentricity-based max pooling to account for the loss of detail in peripheral vision. We apply our model to predict human fixations in a visual search task for objects in real-world scenes. Unlike data-driven models, our model does not require training on large eye movement datasets and can generalize to any set of natural images and targets. We evaluate the generalization capability of our model by demonstrating its results on two publicly available visual search datasets, Ehinger and COCO-search18, without any further training on those datasets. Our model outperforms or performs comparably to data-driven models that are directly trained on human eye movement datasets.

A Few-Shot Defect Detection Method for Transmission Lines Based on Meta-Attention and Feature Reconstruction

In tasks of transmission line defect detection, traditional object detection algorithms are ineffective, with few training samples of defective components. Meta-learning uses multi-task learning as well as fine-tuning to learn common features in different tasks, which has the ability to adapt to new tasks quickly, shows good performance in few-shot object detection, and has good generalization in new tasks. For this reason, we proposed a few-shot defect detection method (Meta PowerNet) with a Meta-attention RPN and Feature Reconstruction Module for transmission lines based on meta-learning. First, in the stage of region proposal, a new region proposal network (Meta-Attention Region Proposal Network, MA-RPN) is designed to fuse the support set features and the query set features to filter the noise in anchor boxes. In addition, it has the ability to focus on the subtle texture features of smaller-sized objects by fusing low-level features from the query set. Second, in the meta-feature construction stage, we designed a meta-learner with the defect feature reconstruction module as the core to capture and focus on the defect-related feature channels. The experimental results show that under the condition, there are only 30 training objects for various types of component defects. The method achieves 72.5% detection accuracy for component defects, which is a significant improvement compared with other mainstream few-shot object detection. Meanwhile, the MA-RPN designed in this paper can be used in other meta-learning object detection models universally.

Estimation method and research of fruit glycemic load index based on the fusion SE module Faster R-CNN

In recent years, with the close connection between the field of artificial intelligence and life, people's quality of life has been improved, and food choices have increased. However, how to understand one's own dietary structure and sugar intake and obtain the correct glycemic load index reference has become a more concerned issue for diabetic patients. This paper studies how to measure the dietary intake of diabetic patients. Our purpose is to identify fruits with high sweetness and low glycemic load value, to help diabetic patients give correct and reasonable reference to their daily diet. For better research, we constructed a fruit dataset for studying the daily diet of diabetic patients. At the same time, we improve the Faster R-CNN network for fruit recognition, introduce an attention mechanism module in the feature extraction stage, adjust the anchor aspect ratio of the Region Proposal Network (RPN), and perform a fusion update operation in the fully connected layer. Experiments show that, compared with existing methods, our method im-proves the precision and average recall rate by 2.47% and 3.06%, respectively. Then we designed a volume estimation method, and then obtained the glycemic load index of the fruit, and finally gave an eating opinion based on this value.