Object Detection Applications Research Articles

Real Time Application for Automatic Object and 3D Position Detection and Sorting with Robotic Manipulator

This work deals with the likelihood of merging a 3D sensor into a robotic manipulator, with an objective to automatically detect, track and grasp an object, placing it in another location. To enhance the flexibility and easy functionality of the robot, MATLAB, a versatile and powerful programming language is used to control the robot. For this work, a common industrial task in many factories of pick and place is implemented. A robotic system consisting of an ABB IRB120 robot equipped with a gripper and a 3D Kinect for Windows camera sensor is used. The three-dimensional data acquisition, image processing and some different parameters of the camera are investigated. The information in the image acquired from the camera is used to determine the robot’s working space and to recognize workpieces. This information is then used to calculate the position of the objects. Using this information, an automatic path to grasp an object was designed and developed to compute the possible trajectory to an object in real time. To be able to detect the workpieces, object recognition techniques are applied using available algorithms in MATLAB’s Computer Vision Toolbox and Image Acquisition Toolbox. These give information about the position of the object of interest and its orientation. The information is therefore sent to the robot to create a path through a server-to-client connection over a computer network in real time.

Transformation of Multispectral Data to Quasi-Hyperspectral Data Using Convolutional Neural Network Regression

Hyperspectral (HS) data are proven to be more resourceful compared to multispectral (MS) data for object detection, classification, and several other applications. However, absence of any space-borne HS sensor since 2017, which can provide open-source data with global coverage, and high cost and limited obtainability of airborne sensors-based images limit the use of HS data. Transformation of readily available MS data into quasi-HS data can be a feasible solution for this issue. In this article, we propose the use of convolutional neural network regression (CNNR), a deep learning-based algorithm, for MS (i.e., Landsat 7/8) to quasi-HS (i.e., quasi-Hyperion) data transformation. The proposed CNNR model is compared with the existing pseudo-HS image transformation algorithm (PHITA), a simple linear model [i.e., stepwise linear regression (SLR)], and a nonlinear modeling approach [i.e., support vector regression (SVR)] by evaluating the quality of the quasi-Hyperion data. Contrary to these existing and simple models, the proposed CNNR model has the added advantage of utilizing deep learning-based spectral-spatial features for MS to quasi-HS data transformation through regression-based nonlinear modeling. Different statistical metrics are calculated to compare each band's reflectance values as well as spectral reflectance curve of each pixel of the quasi-Hyperion data with that of the original Hyperion data. The developed models and generated quasi-Hyperion data are also evaluated with application to crop classification. Analyzing the results of all the experiments, it is evident that CNNR model is more efficient compared to PHITA, SLR, and SVR in creating the quasi-Hyperion data and this transformed data are proven to be resourceful for crop classification application. The proposed CNNR model-based MS to quasi-HS data transformation approach can be used as a viable alternative for different applications in the absence of original HS images.

Tomato Diseases and Pests Detection Based on Improved Yolo V3 Convolutional Neural Network.

Tomato is affected by various diseases and pests during its growth process. If the control is not timely, it will lead to yield reduction or even crop failure. How to control the diseases and pests effectively and help the vegetable farmers to improve the yield of tomato is very important, and the most important thing is to accurately identify the diseases and insect pests. Compared with the traditional pattern recognition method, the diseases and pests recognition method based on deep learning can directly input the original image. Instead of the tedious steps such as image preprocessing, feature extraction and feature classification in the traditional method, the end-to-end structure is adopted to simplify the recognition process and solve the problem that the feature extractor designed manually is difficult to obtain the feature expression closest to the natural attribute of the object. Based on the application of deep learning object detection, not only can save time and effort, but also can achieve real-time judgment, greatly reduce the huge loss caused by diseases and pests, which has important research value and significance. Based on the latest research results of detection theory based on deep learning object detection and the characteristics of tomato diseases and pests images, this study will build the dataset of tomato diseases and pests under the real natural environment, optimize the feature layer of Yolo V3 model by using image pyramid to achieve multi-scale feature detection, improve the detection accuracy and speed of Yolo V3 model, and detect the location and category of diseases and pests of tomato accurately and quickly. Through the above research, the key technology of tomato pest image recognition in natural environment is broken through, which provides reference for intelligent recognition and engineering application of plant diseases and pests detection.

Plants meet machines: Prospects in machine learning for plant biology

Plants meet machines: Prospects in machine learning for plant biology

Simple weakly supervised deep learning pipeline for detecting individual red-attacked trees in VHR remote sensing images

ABSTRACT After an attack the by pine wood nematode, pine tree needles turn red. Using convolutional neural networks (CNNs) based object detection methods, machines can detect red-attacked trees. However, most deep learning object detection algorithms (such as Faster R-CNN and YOLO among others) often require a large number of labelled training datasets, where in each image every object must be given a bounding box label. To increase the cost-effectiveness of this process, we propose a simple yet efficient weakly supervised processing pipeline, based on class activation maps to locate the target. Unlike object detection methods that require bounding-box-labelled data for training, the proposed pipeline only needs image-level-labelled data. Using the proposed pipeline, we could achieve an average precision (AP) of 91.82% on test dataset. Comparing with sliding window-based method which achieves an average precision (AP) of 89.95%, our method not only gets a better AP but also runs faster than sliding window-based pipeline. This result not only indicates that the pipeline is a highly effective one but also demonstrates that image-level-labelled aerial images can be used for the detection of red-attacked tree. The proposed method should also find use in other object detection applications in the field of remote sensing.

A dedicated hardware accelerator for real-time acceleration of YOLOv2

In recent years, dedicated hardware accelerators for the acceleration of the convolutional neural network (CNN) have been extensively studied. Although many studies have presented efficient designs on FPGAs for image classification neural network models such as AlexNet and VGG, there are still little implementations for CNN-based object detection applications. This paper presents an OpenCL-based high-throughput FPGA accelerator for the YOLOv2 object detection algorithm on Arria-10 GX1150 FPGA. The proposed hardware architecture adopts a scalable pipeline design to support multi-resolution input image and full 8-bit fixed-point datapath to improve hardware resource utilization. Layer fusion technology that merges the convolution, batch normalization and Leaky-ReLU is also developed to avoid transmission of intermediate data between FPGA and external memory. Experimental results show that the final design achieves a peak throughput of 566 GOP/s under the working frequency of 190 MHz. The accelerator can execute YOLOv2 inference computation ( $$288\times 288$$ resolution) and tiny YOLOv2 ( $$416\times 416$$ resolution) at the speed of 35 and 71 FPS, respectively.

Overcoming Occlusion in the Automotive Environment—A Review

Accurate and consistent vulnerable road user detection remains one of the most challenging perception tasks for autonomous vehicles. One of the most complex outstanding issues is partial occlusion, where a sensor has only a partial view of the target object due to a foreground object that partially obscures the target. A review of occlusion detection and handling solutions for the automotive environment is presented by this research. This article first discusses object detection by the human visual system, provides an overview of occlusion reasoning in computer vision, presents a summary of occlusion handling strategies in pedestrian, vehicle and object detection applications in the automotive environment. A selection of the remaining challenges to achieving the required level of object detection performance for safe autonomous driving are also discussed.

Design of Deep Learning VLIW Processor for Image Recognition

In order to adapt the application demands of high resolution images recognition and efficient processing of localization in aviation and aerospace fields, and to solve the problem of insufficient parallelism in existing researches, an extensible multiprocessor cluster deep learning processor architecture based on VLIW is designed by optimizing the computation of each layer of deep convolutional neural network model. Parallel processing of feature maps and neurons, instruction level parallelism based on very long instruction word (VLIW), data level parallelism of multiprocessor clusters and pipeline technologies are adopted in the design. The test results based on FPGA prototype system show that the processor can effectively complete the image classification and object detection applications. The peak performance of processor is up to 128 GOP/s when it operates at 200 MHz. For selecting benchmarks, the processor speed is about 12X faster than CPU and 7X faster than GPU at least. Comparing with the results of the software framework, the average error of the test accuracy of the processor is less than 1%.

Local Feature Descriptor Indexing for Image Matching and Object Detection in Real-Time Applications

Local feature extraction and description algorithms can be used to address a large variety of computer vision problems, including pattern recognition, frame stitching and 3D reconstruction. One of the most computationally expensive and time-consuming stages of any method based on local features is keypoint matching. This paper discusses possible ways to optimize matching for frame alignment and object detection applications using Vantage-Point tree indexing to optimize pairwise keypoint matching, and image keypoint graph indexing to optimize pose estimation.

Exploring Deep Learning-Based Architecture, Strategies, Applications and Current Trends in Generic Object Detection: A Comprehensive Review

Object detection is a fundamental but challenging issue in the field of generic image analysis; it plays an important role in a wide range of applications and has been receiving special attention in recent years. Although there are enomerous methods exist, an in-depth review of the literature concerning generic detection remains. This paper provides a comprehensive survey of recent advances in visual object detection with deep learning. Covering about 300 publications that we survey 1) region proposal-based object detection methods such as R-CNN, SPPnet, Fast R-CNN, Faster R-CNN, Mask RCN, RFCN, FPN, 2) classification/regression base object detection methods such as YOLO(v2 to v5), SSD, DSSD, RetinaNet, RefineDet, CornerNet, EfficientDet, M2Det 3) Some latest detectors such as, relation network for object detection, DCN v2, NAS FPN. Moreover, five publicly available benchmark datasets and their standard evaluation metrics are also discussed. We mainly focus on the application of deep learning architectures to five major applications, namely Object Detection in Surveillance, Military, Transportation, Medical, and Daily Life. In the survey, we cover a variety of factors affecting the detection performance in detail, such as i) a wide range of object categories and intra-class variations, ii) limited storage capacity and computational power. Finally, we finish the survey by identifying fifteen current trends and promising direction for future research.

Real World Object Detection Dataset for Quadcopter Unmanned Aerial Vehicle Detection

Recent years have shown a noticeable rise in the number of incidents with drones, related to both civilian and military installations. While drone neutralization techniques have become increasingly effective, detection most often relies on professional equipment, which is too expensive to be used for all critical nodes and applications. Therefore, there is a need for drone detection systems that could work on low performance hardware. Its critical component consists of an object detection system. In this article, we introduce a new object detection dataset, built entirely to train computer vision based object detection machine learning algorithms for a task of binary object detection to enable automated, industrial camera based detection of multiple drone objects using camera feed. The dataset expands existing multiclass image classification and object detection datasets (ImageNet, MS-COCO, PASCAL VOC, anti-UAV) with a diversified dataset of drone images. In order to maximize the effectiveness of the model, real world footage was utilized, transformed into images and hand-labelled to create a custom set of 56821 images and 55539 bounding boxes. Additionally, semi-automated labelling was proposed, tested and proved to be very useful for object detection applications. The dataset was divided into train and test subsets for further processing and used to generate 603 easily deployable Haar Cascades as well as 819 high performing Deep Neural Networks based models. They were used to test different object detection methods to determine the long term feasibility of a large scale drone detection system utilizing machine learning algorithms. The study has shown that Haar Cascade can be used as the Minimum Viable Product model for mediocre performance but fails to scale up effectively for a larger dataset compared to the Deep Neural Network model.

Circle Object Detection Of A Moveable Vision Robot Using Hough Circle Method

The object detection system is very important in an automatic mobile robot system. The functions of this system for a navigation robot are to find objects and also at the same time to avoid obstacle. Detecting objects here including the shape, size, and color of object. Various implementations of robot detection have been made in various fields, especially in industrial field. In this study, a circular object detection application system was developed using the Hough Circle Transformation method. Circle object detection system developed in the following stages: Image processing (capturing video from the camera, BRG color conversion to Gray scale, and Gaussian Blur); the application of Hough Circle Transformation; and calculation of diameter and distance of circle objects. The test results show that the system works well to detect circle objects in various variations of the object given. The system can also detect the distance of circle objects with an average error of 8.17%. By contrast, upon measuring circle object diameter, the system can measure with an average error of 5.55%.

R-CNN Derin Öğrenme Mimarisi ile Vücut Kondisyon Skoru (VKS) Alanı Bölütlenmesi ve Sınıflandırması

Body condition score (BCS) is based on scoring of dairy cattle from 1 to 5 according to the appearance of animals. BCS is a subjective method based on assessing of subcutaneous fat thickness on the regions in back, waist and coccyx regions in cattle and the bone spurs in the pelvic region by visual inspection and palpation method. BSC of animals in among the most important indicator of whether the needs of animals are met in livestock enterprises. In general, BCS values are determined by a method based on expert knowledge and determined by observation. If the animal is above or below the desired BCS, at this stage, diseases resulting from metabolic problems, low yield or animal losses may occur. With the regular control of this situation, the profitability of the enterprise may increase with healthier animals. For this purpose, in this study, it is aimed to segment the required regions and to classify the segmented regions in order to perform BCS. Images taken from dairy cattle were trained with the R-CNN architecture used in object detection applications, which are among the Convolutional Neural Networks (CNN) architectures. Of the 184 images, 75% (138) were used for training and 25% (46) were used for testing. During the training phase, the regions where BSC could be conducted from the raw images were labeled and these regions were learned. Then, the segmentation of the correct regions from the new images to the system was tested. Pre-trained networks were utilized to increase system success. For the classification of the segmented regions, the CNN network trained with AlexNet architecture was used. When the overall success of the system was evaluated, the AlexNet network correctly segmented 40 of the 46 raw test images, and the AlexNet CNN network correctly classified 28 of them and provided 60.86% overall success. The VGG16 network correctly segmented 42 of the 46 raw test images, and the AlexNet CNN network correctly classified 30 of them, achieving 65.21% overall success On the other hand, The VGG19 network correctly segmented 43 of the 46 raw test images, and the AlexNet CNN network correctly classified 31 of them, achieving 67.39% overall success.

Implementation of an Obstacle Recognition System for the Blind

The blind encounter commuting risks, such as failing to recognize and avoid obstacles while walking, but protective support systems are lacking. Acoustic signals at crosswalk lights are activated by button or remote control; however, these signals are difficult to operate and not always available (i.e., broken). Bollards are posts installed for pedestrian safety, but they can create dangerous situations in that the blind cannot see them. Therefore, we proposed an obstacle recognition system to assist the blind in walking safely outdoors; this system can recognize and guide the blind through two obstacles (crosswalk lights and bollards) with image training from the Google Object Detection application program interface (API) based on TensorFlow. The recognized results notify the blind through voice guidance playback in real time. The single shot multibox detector (SSD) MobileNet and faster region-convolutional neural network (R-CNN) models were applied to evaluate the obstacle recognition system; the latter model demonstrated better performance. Crosswalk lights were evaluated and found to perform better during the day than night. They were also analyzed to determine if a client could cross at a crosswalk, while the locations of bollards were analyzed by algorithms to guide the client by voice guidance.

Smartphone‐based object recognition with embedded machine learning intelligence for unmanned aerial vehicles

AbstractExisting artificial intelligence solutions typically operate in powerful platforms with high computational resources availability. However, a growing number of emerging use cases such as those based on unmanned aerial systems (UAS) require new solutions with embedded artificial intelligence on a highly mobile platform. This paper proposes an innovative UAS that explores machine learning (ML) capabilities in a smartphone‐based mobile platform for object detection and recognition applications. A new system framework tailored to this challenging use case is designed with a customized workflow specified. Furthermore, the design of the embedded ML leverages TensorFlow, a cutting‐edge open‐source ML framework. The prototype of the system integrates all the architectural components in a fully functional system, and it is suitable for real‐world operational environments such as seek and rescue use cases. Experimental results validate the design and prototyping of the system and demonstrate an overall improved performance compared with the state of the art in terms of a wide range of metrics.

Trajectory Object Detection using Deep Learning Algorithms

Video surveillance data in smart cities needs to analyze a large amount of video footage in order to locate the people who are violating the traffic rules. The fact is that it is very easy for the human being to recognize different objects in images and videos. For a computer program this is quite a difficult task. Hence there is a need for visual big data analytics which involves processing and analyzing large scale visual data such as images or videos. One major application of trajectory object detection is the Intelligent Transport Systems (ITS). Vehicle type detection, tracking and classification play an important role in ITS. In order to analyze huge amount of video footage deep learning algorithms have been deployed. The main phase of vehicle type detection includes annotating the data, training the model and validating the model. The problems and challenges in identifying or detecting type of vehicle are due to weather, shadows, blurring effect, light condition and quality of the data. In this paper deep learning algorithms such as Faster R CNN and Mask R CNN and Frameworks like YOLO were used for the object detection. Dataset (different types of vehicle pictures in video format) were collected both from in-house premises as well as from the Internet to detect and recognize the type of vehicles which are common in traffic systems. The experimental results show that among the three approaches used the Mask R CNN algorithm is found to be more efficient and accurate in vehicle type detection.

A comparison of shape-based matching with deep-learning-based object detection

Abstract Matching, i. e. determining the exact 2D pose (e. g., position and orientation) of objects, is still one of the key tasks in machine vision applications like robot navigation, measuring, or grasping an object. There are many classic approaches for matching, based on edges or on the pure gray values of the template. In recent years, deep learning has been utilized mainly for more difficult tasks where the objects of interest are from many different categories with high intra-class variations and classic algorithms are failing. In this work, we compare one of the latest deep-learning-based object detectors with classic shape-based matching. We evaluate the methods both on a matching dataset as well as an object detection dataset that contains rigid objects and is thus also suitable for shape-based matching. We show that for datasets of this type, where rigid objects appear with rigid transformations, shape-based matching still outperforms recent object detectors regarding runtime, robustness, and precision if only a single template image per object is used. On the other hand, we show that for the application of object detection, the deep-learning-based approach outperforms the classic approach if annotated data is used for training. Ultimately, the choice of the best suited approach depends on the conditions and requirements of the application.

Kankanet: An artificial neural network-based object detection smartphone application and mobile microscope as a point-of-care diagnostic aid for soil-transmitted helminthiases.

BackgroundEndemic areas for soil-transmitted helminthiases often lack the tools and trained personnel necessary for point-of-care diagnosis. This study pilots the use of smartphone microscopy and an artificial neural network-based (ANN) object detection application named Kankanet to address those two needs.Methodology/Principal findingsA smartphone was equipped with a USB Video Class (UVC) microscope attachment and Kankanet, which was trained to recognize eggs of Ascaris lumbricoides, Trichuris trichiura, and hookworm using a dataset of 2,078 images. It was evaluated for interpretive accuracy based on 185 new images. Fecal samples were processed using Kato-Katz (KK), spontaneous sedimentation technique in tube (SSTT), and Merthiolate-Iodine-Formaldehyde (MIF) techniques. UVC imaging and ANN interpretation of these slides was compared to parasitologist interpretation of standard microscopy.Relative to a gold standard defined as any positive result from parasitologist reading of KK, SSTT, and MIF preparations through standard microscopy, parasitologists reading UVC imaging of SSTT achieved a comparable sensitivity (82.9%) and specificity (97.1%) in A. lumbricoides to standard KK interpretation (97.0% sensitivity, 96.0% specificity). The UVC could not accurately image T. trichiura or hookworm. Though Kankanet interpretation was not quite as sensitive as parasitologist interpretation, it still achieved high sensitivity for A. lumbricoides and hookworm (69.6% and 71.4%, respectively). Kankanet showed high sensitivity for T. trichiura in microscope images (100.0%), but low in UVC images (50.0%).Conclusions/SignificanceThe UVC achieved comparable sensitivity to standard microscopy with only A. lumbricoides. With further improvement of image resolution and magnification, UVC shows promise as a point-of-care imaging tool. In addition to smartphone microscopy, ANN-based object detection can be developed as a diagnostic aid. Though trained with a limited dataset, Kankanet accurately interprets both standard microscope and low-quality UVC images. Kankanet may achieve sensitivity comparable to parasitologists with continued expansion of the image database and improvement of machine learning technology.

Efficient multi-scale non-sub-sampled shearlet fusion system based on modified central force optimization and contrast enhancement

Infrared, visible, and medical image fusion systems are mandatory solutions for obtaining more spatial and more spectral information in a single image for efficient object detection and medical diagnosis applications. In this paper, an optimized fusion system for multi-modality images based on the Non-sub-Sampled Shearlet Transform (NSST) with Modified Central Force Optimization (MCFO), histogram matching, and local contrast enhancement is presented. The proposed multi-modality image fusion system consists of four stages. The first stage is the image registration and then histogram matching of one of the images to the other to allow the same dynamic range for both images to minimize the fusion artifacts. The NSST is used after that to decompose the images to be fused into their coefficients. The NSST provides a better sparse image representation of highly localized coefficients, anisotropic directionality, and reduced pseudo-Gibbs artifacts. After that, the MCFO technique is used to determine the optimum decomposition level and the optimum gain parameters for the best fusion of coefficients based on certain constraints. Finally, an additional contrast enhancement process is applied on the fused image to enhance its visual quality and reinforce details. The proposed fusion system is subjectively and objectively evaluated with different fusion quality metrics including average gradient, local contrast, standard deviation, edge intensity, entropy, Peak Signal-to-Noise Ratio (PSNR), and Qab/f. Real infrared, visible, and medical datasets of different modalities are used to test the proposed system. Experimental results demonstrate that the proposed system achieves a superior performance with higher image quality, higher evaluation metrics values, and much more details in images. These characteristics help for more accuracy in applications such as object detection and medical diagnosis.

Integrating computer vision and non-linear optimization for automated deformable registration of 3D medical images

Deformable image registration (DIR), the process of estimating and applying non-linear transforms to spatially align sets of two or more images, is a challenging task with many important clinical applications including kinetic analysis, cancer treatment targeting, and evaluation of treatment response. Current techniques use non-linear optimizations only to reach a local minimum and not a globally optimal solution, limiting application to cases with small spatial displacements. Various semi-dense feature-based methods drawing inspiration from mammalian systems as the basis for 2D visual processing have been implemented for automated wide baseline registration and object detection applications with great success. Extension to the 3D case, which has been shown to enable highly efficient coarse global image search, in theory could be adapted also to allow precise semi-dense global optimization. The algorithm described in this paper, dubbed constrained robust affine feature transform (CRAFT), incorporates paradigms from various computer-vision techniques to combine aspects of the human visual pathway with proven non-linear optimization methods to automate general deformable registration with unprecedented robustness. This hybrid technique is able to estimate registration confidence and can serve as the basis of machine perception of medical images for machine learning.