Visual Object Detection Research Articles

Fruit ripeness identification using transformers

Pattern classification has always been essential in computer vision. Transformer paradigm having attention mechanism with global receptive field in computer vision improves the efficiency and effectiveness of visual object detection and recognition. The primary purpose of this article is to achieve the accurate ripeness classification of various types of fruits. We create fruit datasets to train, test, and evaluate multiple Transformer models. Transformers are fundamentally composed of encoding and decoding procedures. The encoder is to stack the blocks, like convolutional neural networks (CNN or ConvNet). Vision Transformer (ViT), Swin Transformer, and multilayer perceptron (MLP) are considered in this paper. We examine the advantages of these three models for accurately analyzing fruit ripeness. We find that Swin Transformer achieves more significant outcomes than ViT Transformer for both pears and apples from our dataset.

Improving Crowded Object Detection via Copy-Paste

Crowdedness caused by overlapping among similar objects is a ubiquitous challenge in the field of 2D visual object detection. In this paper, we first underline two main effects of the crowdedness issue: 1) IoU-confidence correlation disturbances (ICD) and 2) confused de-duplication (CDD). Then we explore a pathway of cracking these nuts from the perspective of data augmentation. Primarily, a particular copy- paste scheme is proposed towards making crowded scenes. Based on this operation, we first design a "consensus learning" method to further resist the ICD problem and then find out the pasting process naturally reveals a pseudo "depth" of object in the scene, which can be potentially used for alleviating CDD dilemma. Both methods are derived from magical using of the copy-pasting without extra cost for hand-labeling. Experiments show that our approach can easily improve the state-of-the-art detector in typical crowded detection task by more than 2% without any bells and whistles. Moreover, this work can outperform existing data augmentation strategies in crowded scenario.

Single-trial extraction of event-related potentials (ERPs) and classification of visual stimuli by ensemble use of discrete wavelet transform with Huffman coding and machine learning techniques

BackgroundPresentation of visual stimuli can induce changes in EEG signals that are typically detectable by averaging together data from multiple trials for individual participant analysis as well as for groups or conditions analysis of multiple participants. This study proposes a new method based on the discrete wavelet transform with Huffman coding and machine learning for single-trial analysis of evenal (ERPs) and classification of different visual events in the visual object detection task.MethodsEEG single trials are decomposed with discrete wavelet transform (DWT) up to the 4{th} level of decomposition using a biorthogonal B-spline wavelet. The coefficients of DWT in each trial are thresholded to discard sparse wavelet coefficients, while the quality of the signal is well maintained. The remaining optimum coefficients in each trial are encoded into bitstreams using Huffman coding, and the codewords are represented as a feature of the ERP signal. The performance of this method is tested with real visual ERPs of sixty-eight subjects.ResultsThe proposed method significantly discards the spontaneous EEG activity, extracts the single-trial visual ERPs, represents the ERP waveform into a compact bitstream as a feature, and achieves promising results in classifying the visual objects with classification performance metrics: accuracies 93.60pm {6.5}, sensitivities 93.55pm {4.5}, specificities 94.85pm {4.2}, precisions 92.50pm {5.5}, and area under the curve (AUC) 0.93pm {0.3} using SVM and k-NN machine learning classifiers.ConclusionThe proposed method suggests that the joint use of discrete wavelet transform (DWT) with Huffman coding has the potential to efficiently extract ERPs from background EEG for studying evoked responses in single-trial ERPs and classifying visual stimuli. The proposed approach has O(N) time complexity and could be implemented in real-time systems, such as the brain-computer interface (BCI), where fast detection of mental events is desired to smoothly operate a machine with minds.

Enhancing Nanoparticle Detection in Interferometric Scattering (iSCAT) Microscopy Using a Mask R-CNN.

Interferometric scattering microscopy (iSCAT) is a label-free optical microscopy technique that enables imaging of individual nano-objects such as nanoparticles, viruses, and proteins. Essential to this technique is the suppression of background scattering and identification of signals from nano-objects. In the presence of substrates with high roughness, scattering heterogeneities in the background, when coupled with tiny stage movements, cause features in the background to be manifested in background-suppressed iSCAT images. Traditional computer vision algorithms detect these background features as particles, limiting the accuracy of object detection in iSCAT experiments. Here, we present a pathway to improve particle detection in such situations using supervised machine learning via a mask region-based convolutional neural network (mask R-CNN). Using a model iSCAT experiment of 19.2 nm gold nanoparticles adsorbing to a rough layer-by-layer polyelectrolyte film, we develop a method to generate labeled datasets using experimental background images and simulated particle signals and train the mask R-CNN using limited computational resources via transfer learning. We then compare the performance of the mask R-CNN trained with and without inclusion of experimental backgrounds in the dataset against that of a traditional computer vision object detection algorithm, Haar-like feature detection, by analyzing data from the model experiment. Results demonstrate that including representative backgrounds in training datasets improved the mask R-CNN in differentiating between background and particle signals and elevated performance by markedly reducing false positives. The methodology for creating a labeled dataset with representative experimental backgrounds and simulated signals facilitates the application of machine learning in iSCAT experiments with strong background scattering and thus provides a useful workflow for future researchers to improve their image processing capabilities.

One-Stage Methods of Computer Vision Object Detection to Classify Carious Lesions from Smartphone Imaging

The current study aimed to implement and validate an automation system to detect carious lesions from smartphone images using different one-stage deep learning techniques. 233 images of carious lesions were captured using a smartphone camera system at 1432 × 1375 pixels, then classified and screened according to a visual caries classification index. Following data augmentation, the YOLO v5 model for object detection was used. After training the model with 1452 images at 640 × 588 pixel resolution, which included the ones that were created via image augmentation, a discrimination experiment was performed. Diagnostic indicators such as true positive, true negative, false positive, false negative, and mean average precision were used to analyze object detection performance and segmentation of systems. YOLO v5X and YOLO v5M models achieved superior performance over the other models on the same dataset. YOLO v5X’s mAP was 0.727, precision was 0.731, and recall was 0.729, which was higher than other models of YOLO v5, which generated 64% accuracy, with YOLO v5M producing slightly inferior results. Overall mAPs of 0.70, precision of 0.712, and recall of 0.708 were achieved. Object detection through the current YOLO models was able to successfully extract and classify regions of carious lesions from smartphone photographs of in vitro tooth specimens with reasonable accuracy. YOLO v5M was better fit to detect carious microcavitations while YOLO v5X was able to detect carious changes without cavitation. No single model was capable of adequately diagnosing all classifications of carious lesions.

Wave Detection and Tracking Within a Rotating Detonation Engine Through Object Detection

As the operational time window of experimental rotating detonation engines (RDEs) is expanded and the technology matures toward integration within gas turbines, monitoring techniques must evolve to offer computationally efficient and highly time-resolved diagnostics. A computer vision object detection methodology that seeks to reduce data processing time and calculate wave velocity within drastically reduced time intervals as compared to traditional high-frame-rate RDE images analysis techniques is proposed. The adapted you-only-look-once object detection network is trained to detect individual detonation waves within single down-axis RDE images. The wave location and rotational direction detected within a frame are tracked through a series of high-speed images to calculate the frame-to-frame wave velocity with the time-step resolution of across a series of frames. The analysis of the annotation box size and image linearization effects is presented, demonstrating the lowest frame-to-frame velocity total uncertainty of and the highest classification speed of 9.5 frames per second using linearized images. Linearized images “unwrap” the RDE annulus pixel region to a reduced image size. This new method offers great reductions in data processing times and unsteady detonation behavior insight at intervals more comparable to the timescales of detonation wave interactions via the application of machine learning to experimental RDE data.

6D Object Localization in Car-Assembly Industrial Environment

In this work, a visual object detection and localization workflow integrated into a robotic platform is presented for the 6D pose estimation of objects with challenging characteristics in terms of weak texture, surface properties and symmetries. The workflow is used as part of a module for object pose estimation deployed to a mobile robotic platform that exploits the Robot Operating System (ROS) as middleware. The objects of interest aim to support robot grasping in the context of human-robot collaboration during car door assembly in industrial manufacturing environments. In addition to the special object properties, these environments are inherently characterised by cluttered background and unfavorable illumination conditions. For the purpose of this specific application, two different datasets were collected and annotated for training a learning-based method that extracts the object pose from a single frame. The first dataset was acquired in controlled laboratory conditions and the second in the actual indoor industrial environment. Different models were trained based on the individual datasets and a combination of them were further evaluated in a number of test sequences from the actual industrial environment. The qualitative and quantitative results demonstrate the potential of the presented method in relevant industrial applications.

Visual Object Detection and Tracking for Internet of Things Devices Based on Spatial Attention Powered Multidomain Network

Internet of Things (IoT) has brought changes in many fields by joining physical space with the cyber space. The IoT devices are becoming increasingly complex. With the rapid deployment of cameras, tasks in IoT like visual information are more important, but IoT devices have limited computing resources, including power, computing ability, storage, etc. Some tasks that might be perfectly normal to perform on a computer would be rather challenging on an IoT device. Therefore, how to maintain acceptable performance while minimizing resources is becoming a more consequential part in IoT. In this article, we aim to solve the problem of object detection and tracking in IoT while minimizing resources. The traditional algorithms need to use a convolutional neural network (CNN) to identify different objects in each frame, and then determine the tracking target from identified objects, which typically requires a lot of computing resources. By incorporating spatial attention, and multidomain network, we proposed a novel algorithm named as spatial attention powered multidomain network (SA-MDNet). By adding the spatial attention mechanism to the original MDNet model, and using multiclass cross-entropy loss, we are able to distinguish the background and the target in different video sequences effectively and efficiently. This novel algorithm achieves similar performance on the OTB 50/100/2013 data sets compared to several state-of-art models, while uses only a fraction of the memory compared to MDNet.

Rethinking unsupervised texture defect detection using PCA

In recent times, deep learning frameworks have become the preferred solution for visual object detection tasks, including texture defect detection. However, in real-world industrial applications, such as operation and maintenance of gas-insulated switchgear (GIS) equipment, collecting a large number of samples in advance is difficult because opening GIS equipment affects its normal operation; moreover, only four real-world GIS equipment defect samples are available currently. Most deep learning methods perform poorly when trained using a small number of labelled samples or when there is a lack of computational resources. In addition, the features extracted by multi-layer convolution and down-sampling are often unsuitable for small-size and low-contrast defects. However, the structure of the repetitive textured surface is simple and easy to model. Hence, this paper proposes a texture-defect detection method using principal components analysis (PCA) and histogram-based outlier score (HBOS) that requires only a small number of unlabelled samples and low computational complexity. First, PCA is utilized to extract features from the training patches to enhance the difference between the defect and background regions. Then, non-saliency suppression is applied for feature fusion to obtain the saliency map. Finally, the saliency value is converted to the defect score, and the defect is located by HBOS. Experiments on two public datasets and a GIS simulation equipment demonstrate that the proposed method is suitable for product surface defect detection and operation and maintenance in power distribution equipment. Moreover, compared with traditional texture feature-based and deep learning methods, the proposed method yields better results for small-size and low-contrast defects with only four unlabelled training samples per class.

Visual Object Detection for Privacy-Preserving Federated Learning

Visual object detection is a computer vision technique based on deep learning. However, due to privacy issues, it is highly challenging to build an object detection model according to the current methods on the centrally stored training dataset. Federated learning is an effective approach to solving the challenge of training data collection by using distributed training. We propose FedVisionBC, a blockchain-based federated learning system for visual object detection that addresses the challenges of single point of failure, model poisoning attacks, and membership inference attacks in traditional federated learning. In the FedVisionBC system, we set up an aggregation node and a verification node instead of a central server to solve the single point of failure problem. We establish a security mechanism that uses encryption techniques, verification nodes, and smart contracts to resist model poisoning attacks. Experimental results show that FedVisionBC can accomplish the object detection task when the percentage of malicious clients is less than 60%. We also propose a new algorithm, ADPFedAvg, to prevent membership inference attacks, which relies on user-level differential privacy technology and the federal average algorithm. Experimental results indicate that ADPFedAvg can achieve a large-scale visual object detection model with differential privacy protection, while only a negligible cost in predictive accuracy.

Ensemble Based Learning with Accurate Motion Contrast Detection

Recent developments in computer vision applications have enabled detection of significant visual objects in video streams. Studies quoted in literature have detected objects from video streams using Spatiotemporal Particle Swarm Optimization (SPSOM) and Incremental Deep Convolution Neural Networks (IDCNN) for detecting multiple objects. However, the study considered optical flows resulting in assessing motion contrasts. Existing methods have issue with accuracy and error rates in motion contrast detection. Hence, the overall object detection performance is reduced significantly. Thus, consideration of object motions in videos efficiently is a critical issue to be solved. To overcome the above mentioned problems, this research work proposes a method involving ensemble approaches to and detect objects efficiently from video streams. This work uses a system modeled on swarm optimization and ensemble learning called Spatiotemporal Glowworm Swarm Optimization Model (SGSOM) for detecting multiple significant objects. A steady quality in motion contrasts is maintained in this work by using Chebyshev distance matrix. The proposed system achieves global optimization in its multiple object detection by exploiting spatial/temporal cues and local constraints. Its experimental results show that the proposed system scores 4.8% in Mean Absolute Error (MAE) while achieving 86% in accuracy, 81.5% in precision, 85% in recall and 81.6% in F-measure and thus proving its utility in detecting multiple objects.

An Enhancement Process for Multi-focus Images Resulted from Image Fusion using qshiftN DTCWT and MPCA in Multiresolution Domain

Images having different focus values are combined using image fusion methodologies. The fused image shows superiority in the quality and highly informative than compared o any other multifocus images. The fused image would show its suitability for visual perception object detection due to its high quality than compared to multifocus images. Among the various quality parameters of the fused image, directional invariance and shift invariance significantly influence the fused image quality. The traditional image fusion methods using wavelets severely suffers with poor invariance and lack of directionality. An efficient image fusion method is developed using DTCWT with qshiftN and MPCA algorithms in the multiresolution (MR) domain. Multifocus input images are decomposed into high and low frequency components using MR algorithm. The decomposed frequency components of the input images are fused using DTCWT with qshiftN algorithm. The proposed fusion algorithm preserves both sift invariance and directional properties of the multifocus source image. Lastly, the fused image is processed using MPCA algorithm to enhance the features. The proposed methodology is assessed using various multifocus images and the evaluated metrics are contrasted with technologically advanced algorithms reported recently. The statistical metrics evaluated for the proposed method for different multifocus images shows superior properties than compared to the recently reported multifocus image fusion algorithms.

Cloud Computing Based Network Analysis in Smart Healthcare System with Neural Network Architecture

The recent progressions in Artificial Intelligence (AI), the Internet of Things (IoT), and cloud computing transformed the traditional healthcare system into a smart healthcare system. Medical services can be improved through the incorporation of key technologies namely AI and IoT. The convergence of AI and IoT renders several openings in the healthcare system. In machine learning, deep learning can be considered a renowned topic with a wide range of applications like biomedicine, computer vision, speech recognition, drug discovery, visual object detection, natural language processing, disease prediction, bioinformatics, etc. Among these applications, medical science-related and health care applications were raised dramatically. This study develops a Cloud computing-based network analysis in the smart healthcare systems with neural network (CCNA-SHSNN) architecture. The presented CCNA-SHSNN technique assists in the decision-making process of the healthcare system in a real time cloud environment. For data pre-processing, the CCNA-SHSNN technique uses a normalization approach. Secondly, the CCNA-SHSNN technique applies the autoencoder (AE) model for healthcare data classification in the CC platform. At last, the gravitational search algorithm (GSA) is used for hyperparameter optimization of the AE model. The experimental outcomes are determined on a benchmark dataset and the outcomes signify the outperforming efficiency of the CCNA-SHSNN technique compared to existing techniques.

Domain Contrast for Domain Adaptive Object Detection

Despite of the substantial progress of visual object detection, models trained in one video domain often fail to generalize well to others due to the change of camera configurations, lighting conditions, and object person views. In this paper, we present Domain Contrast (DC), a simple yet effective approach inspired by contrastive learning for training domain adaptive detectors. DC is deduced from the error bound minimization perspective of a transferred model, and is implemented with cross-domain contrast loss which is plug-and-play. By minimizing cross-domain contrast loss, DC transfers detectors across domains while naturally alleviating the class imbalance issue in the target domain. DC can be applied at either image level or region level, consistently improving detectors’ discriminability while maintaining the transferability. Extensive experiments on commonly used benchmarks show that DC improves the baseline and state-of-the-art by significant margins, while demonstrating great potential for large domain divergence. Code is released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/PhoneSix/Domain-Contrast</uri> .

A meta-fusion RCNN network for endoscopic visual bladder lesions intelligent detection.

This study investigates a visual object detection technology in order to help doctors diagnose bladder lesions with endoscopy. A new object detection approach based on deep learning is presented, which derived from the cascade R-CNN and extended the ability of network for adapting insufficient endoscopic lesions samples when training a deep neural network. We propose a feature adaptive fusion model to increase the network's mobility and reduce the possibility of overfitting problems, and use task adaptation meta-learning approach to train the feature fusion process of the entire model and the target network update process in order to complete the task-adaptive classification and detection. The new model has been evaluated on the challenging object detection data set Pascal VOC and its converted format of Microsoft COCO, and the results show that the performance of our proposed method is superior to the original method. Therefore, we apply the proposed method to a custom bladder lesions data set to solve the auxiliary detection problem in the intelligent diagnosis of bladder lesions and demonstrated the effectiveness.

Faster RCNN based on Feature Fusion and its Application in Steel Surface Defect Detection

Surface defects are an inevitable problem in steel production and processing. To solve the problems of error detection, leakage detection and low accuracy of traditional manual methods for steel surface defect detection, this paper proposes a steel surface defect detection algorithm based on Faster RCNN, a classical model in the field of computer vision object detection. The proposed model uses ResNet50 as its backbone and introduces FPN to fuse the multi-layer feature maps of the backbone to improve the detection capability for defects of different scales. The experimental results on NEU-DET dataset show that the proposed model in this paper has different detection accuracies for six different types of defects in the dataset, and the overall mAP reaches 0.708. In addition, the model execution speed reaches 43.3 frame/s, which meets the applications in industrial scenarios.

Face mask detection and social distance monitoring system for COVID-19 pandemic.

Coronavirus triggers several respirational infections such as sneezing, coughing, and pneumonia, which transmit humans to humans through airborne droplets. According to the guidelines of the World Health Organization, the spread of COVID-19 can be mitigated by avoiding public interactions in proximity and following standard operating procedures (SOPs) including wearing a face mask and maintaining social distancing in schools, shopping malls, and crowded areas. However, enforcing the adaptation of these SOPs on a larger scale is still a challenging task. With the emergence of deep learning-based visual object detection networks, numerous methods have been proposed to perform face mask detection on public spots. However, these methods require a huge amount of data to ensure robustness in real-time applications. Also, to the best of our knowledge, there is no standard outdoor surveillance-based dataset available to ensure the efficacy of face mask detection and social distancing methods in public spots. To this end, we present a large-scale dataset comprising of 10,000 outdoor images categorized into a binary class labeling i.e., face mask, and non-face masked people to accelerate the development of automated face mask detection and social distance measurement on public spots. Alongside, we also present an end-to-end pipeline to perform real-time face mask detection and social distance measurement in an outdoor environment. Initially, existing state-of-the-art single and multi-stage object detection networks are fine-tuned on the proposed dataset to evaluate their performance in terms of accuracy and inference time. Based on better performance, YOLO-v3 architecture is further optimized by tuning its feature extraction and region proposal generation layers to improve the performance in real-time applications. Our results indicate that the presented pipeline performed better than the baseline version, showing an improvement of 5.3% in terms of accuracy.

Re-annotation of training samples for robust maritime object detection

Machine learning and specifically deep learning techniques address many of the issues faced in visual object detection and classification tasks. However, they have the caveat of needing large amounts of annotated training data. In the maritime domain one may encounter objects fairly infrequently, depending on weather and location. This creates an issue of data collection. Areas such as harbors and channels see a lot of traffic, but the ships are of a specific class. Furthermore, the variability of the buoys from region to region and within regions is difficult and expensive to sample. Thus the amount and quality of available data is severely lacking. Furthermore very few publicly available maritime datasets exist In this work, we present a novel approach that detects possible “poor” training samples and automatically re-annotates them, based on the current state of the object detector. We show the applicability of our approach on real-life maritime data and show that the poor annotation quality of the datasets used can be mitigated. We show performance gain with respect to a baseline approach is proportional to the amount of poorly annotated data in the dataset. When 25% of the data is poor we achieve a 5.5%, 13.7%, and 8.0% increase in performance on 3 separate datasets, compared to a baseline model. With 50% noise we reach 58.5%, 18.7% and 94.2% increase respectively. Our approach also allows for the iterative improvement of a given dataset by providing a set of pseudo-annotations to replace the current incorrect ones.

A Self-Learning Object Detection Method Based on Highly Reliable Sample Mining

Visual object detection is an artificial intelligence technique that locates specific objects from images, which is of great significance for practical applications. However, training general object detection models require many manually annotated images, bringing more labour and time cost. In order to improve the adaptability of the object detection model to the data environment changes, this paper proposes a self-learning object detection system based on high-reliability sample mining. We first train a SampleNet that can better mine reliable training samples from unlabeled data. We then use the combination of SampleNet and the basic object detection model to build a complementary residual training framework, continuously improving the sample mining ability and object detection tasks during the training process. The experimental results show that SampleNet can stably provide reliable pseudo samples for model training, and the complementary residual training framework improves the performance of basic object detection tasks.

Floodborne Objects Type Recognition Using Computer Vision to Mitigate Blockage Originated Floods

The presence of floodborne objects (i.e., vegetation, urban objects) during floods is considered a very critical factor because of their non-linear complex hydrodynamics and impacts on flooding outcomes (e.g., diversion of flows, damage to structures, downstream scouring, failure of structures). Conventional flood models are unable to incorporate the impact of floodborne objects mainly because of the highly complex hydrodynamics and non-linear nature associated with their kinematics and accumulation. Vegetation (i.e., logs, branches, shrubs, entangled grass) and urban objects (i.e., vehicles, bins, shopping carts, building waste materials) offer significant materialistic, hydrodynamic and characterization differences which impact flooding outcomes differently. Therefore, recognition of the types of floodborne objects is considered a key aspect in the process of assessing their impact on flooding. The identification of floodborne object types is performed manually by the flood management officials, and there exists no automated solution in this regard. This paper proposes the use of computer vision technologies for automated floodborne objects type identification from a vision sensor. The proposed approach is to use computer vision object detection (i.e., Faster R-CNN, YOLOv4) models to detect a floodborne object’s type from a given image. The dataset used for this research is referred to as the “Floodborne Objects Recognition Dataset (FORD)” and includes real images of floodborne objects blocking the hydraulic structures extracted from Wollongong City Council (WCC) records and simulated images of scaled floodborne objects blocking the culverts collected from hydraulics laboratory experiments. From the results, the Faster R-CNN model with MobileNet backbone was able to achieve the best Mean Average Precision (mAP) of 84% over the test dataset. To demonstrate the practical use of the proposed approach, two potential use cases for the proposed floodborne object type recognition are reported. Overall, the performance of the implemented computer vision models indicated that such models have the potential to be used for automated identification of floodborne object types.