Classical Computer Vision Research Articles

Deep Regularized Discriminative Network

Traditional linear discriminant analysis (LDA) approach discards the eigenvalues which are very small or equivalent to zero, but quite often eigenvectors corresponding to zero eigenvalues are the important dimensions for discriminant analysis. We propose an objective function which would utilize both the principal as well as nullspace eigenvalues and simultaneously inherit the class separability information onto its latent space representation. The idea is to build a convolutional neural network (CNN) and perform the regularized discriminant analysis on top of this and train it in an end-to-end fashion. The backpropagation is performed with a suitable optimizer to update the parameters so that the whole CNN approach minimizes the within class variance and maximizes the total class variance information suitable for both multi-class and binary class classification problems. Experimental results on four databases for multiple computer vision classification tasks show the efficacy of our proposed approach as compared to other popular methods.

Energy-Efficient On-Platform Target Classification for Electric Air Transportation Systems

Due to the predicted rise of Unmanned Aircraft Systems (UAS) in commercial, civil, and military operations, there is a desire to make UASs more energy efficient so they can proliferate with ease of deployment and maximal life per charge. To address current limitations, a three-tiered approach is investigated to mitigate Unmanned Aerial Vehicle (UAV) hover time, reduce network datalink transmission to a ground station, and provide a real-time framework for Sense-and-Avoidance (SAA) target classification. An energy-efficient UAS architecture framework is presented, and a corresponding SAA prototype is developed using commercial hardware to validate the proposed architecture using an experimental methodology. The proposed architecture utilizes classical computer vision methods within the Detection Subsystem coupled with deeply learned Convolutional Neural Networks (CNN) within the Classification Subsystem. Real-time operations of three frames per second are realized enabling UAV hover time and associated energy consumption during SAA processing to be effectively eliminated. Additional energy improvements are not addressed in the scope of this work. Inference accuracy is improved by 19% over baseline COTS models and current non-adaptive, single-stage SAA architectures. Overall, by pushing SAA processing to the edge of the sensors, network offload transmissions and reductions in processing time and energy consumption are feasible and realistic in future battery-powered electric air transportation systems.

Multi-Scale Context Enhanced Network for Monocular Depth Estimation

Monocular depth estimation is a classical computer vision task. At present, most CNN methods cannot effectively combine high-level and low-level features, leading to the loss of details and blurring of boundaries. To solve the problem, we propose a Multi-Scale Context Enhanced Network (MCEN) to learn more abundant context and expand its receptive field for high-accuracy estimation. Our method employs CRE-HRNet (Context and Receptive Enhanced High-Resolution Network) with four branches ranging from low-dimension to high-dimension features to obtain richer contextual information and extract multi-scale features. It then uses RM (Refinement Module) adopting the residual dilated convolution to retains detailed information and improve the receptive field. Finally, non-local block enables our network to capture the longdistance context through its special non-local operation. Experiments with the NYU Depth V2 dataset show its outstanding performance.

Learning 3D face reconstruction from a single sketch

3D face reconstruction from a single image is a classic computer vision problem with many applications. However, most works achieve reconstruction from face photos, and little attention has been paid to reconstruction from other portrait forms. In this paper, we propose a learning-based approach to reconstruct a 3D face from a single face sketch. To overcome the problem of no paired sketch-3D data for supervised learning, we introduce a photo-to-sketch synthesis technique to obtain paired training data, and propose a dual-path architecture to achieve synergistic 3D reconstruction from both sketches and photos. We further propose a novel line loss function to refine the reconstruction with characteristic details depicted by lines in sketches well preserved. Our method outperforms the state-of-the-art 3D face reconstruction approaches in terms of reconstruction from face sketches. We also demonstrate the use of our method for easy editing of details on 3D face models.

A Method for Detection of Small Moving Objects in UAV Videos

Detection of small moving objects is an important research area with applications including monitoring of flying insects, studying their foraging behavior, using insect pollinators to monitor flowering and pollination of crops, surveillance of honeybee colonies, and tracking movement of honeybees. However, due to the lack of distinctive shape and textural details on small objects, direct application of modern object detection methods based on convolutional neural networks (CNNs) shows considerably lower performance. In this paper we propose a method for the detection of small moving objects in videos recorded using unmanned aerial vehicles equipped with standard video cameras. The main steps of the proposed method are video stabilization, background estimation and subtraction, frame segmentation using a CNN, and thresholding the segmented frame. However, for training a CNN it is required that a large labeled dataset is available. Manual labelling of small moving objects in videos is very difficult and time consuming, and such labeled datasets do not exist at the moment. To circumvent this problem, we propose training a CNN using synthetic videos generated by adding small blob-like objects to video sequences with real-world backgrounds. The experimental results on detection of flying honeybees show that by using a combination of classical computer vision techniques and CNNs, as well as synthetic training sets, the proposed approach overcomes the problems associated with direct application of CNNs to the given problem and achieves an average F1-score of 0.86 in tests on real-world videos.

Design and Implementation of a Highly Scalable, Low-Cost Distributed Traffic Violation Enforcement System in Phuket, Thailand

The number of global road traffic accidents is rising every year and remains undesirably high. One of the main reasons for this trend is that, in many countries, road users violate road safety regulations and traffic laws. Despite improvements in road safety legislation, enforcement is still a major challenge in low- and middle-income countries. Information technology solutions have emerged for automated traffic enforcement systems in the last decade. They have been tested on a small scale, but until now, the cost of deployment of these systems is generally too high for nation-wide adoption in low- and middle-income countries that need them the most. We present the architectural design of a traffic violation enforcement system that can optimize the cost of deployment and resource utilization. Based on the proposed architecture, we describe the implementation and deployment of the system, and perform a comparison of two different versions of the video-based enforcement system, one using classical computer vision methods and another using deep learning techniques. Finally, we analyze the impact of the system deployed in Phuket, Thailand from 2017 to the present in terms of local road users’ compliance and the road safety situation. We conclude that the system has had a positive impact on road safety in Phuket at a moderate cost.

The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection

The detection of anomalous structures in natural image data is of utmost importance for numerous tasks in the field of computer vision. The development of methods for unsupervised anomaly detection requires data on which to train and evaluate new approaches and ideas. We introduce the MVTec anomaly detection dataset containing 5354 high-resolution color images of different object and texture categories. It contains normal, i.e., defect-free images intended for training and images with anomalies intended for testing. The anomalies manifest themselves in the form of over 70 different types of defects such as scratches, dents, contaminations, and various structural changes. In addition, we provide pixel-precise ground truth annotations for all anomalies. We conduct a thorough evaluation of current state-of-the-art unsupervised anomaly detection methods based on deep architectures such as convolutional autoencoders, generative adversarial networks, and feature descriptors using pretrained convolutional neural networks, as well as classical computer vision methods. We highlight the advantages and disadvantages of multiple performance metrics as well as threshold estimation techniques. This benchmark indicates that methods that leverage descriptors of pretrained networks outperform all other approaches and deep-learning-based generative models show considerable room for improvement.

A computer vision system for saw blade condition monitoring

Tool condition monitoring is a key component of predictive maintenance in smart manufacturing. Predicting excessive tool wear in machining processes becomes increasingly difficult if different materials need to be processed. We propose a novel computer vision-based system for saw blade condition monitoring that is independent of the processed materials and combines deep learning with classic computer vision. Our approach allows for accurate condition monitoring of blade wear which can further be used for predictive maintenance. Additionally, the system can classify different defect types such as missing blade teeth, thus preventing the production of scrap parts.

Development and Testing on the European Space-Grade BRAVE FPGAs: Evaluation of NG-Large Using High-Performance DSP Benchmarks

The advent of space applications with increased computational requirements has led the space industry to consider innovative chips and avionics architectures for high-performance on-board data processing. In a relatively limited market, the European BRAVE family of Field-Programmable Gate Arrays (FPGAs) offers such novel radiation-hardened solutions. Towards verification, the current work devises and applies a methodology to thoroughly assess the BRAVE FPGAs and their SW tools. The paper focuses on NG-Large, i.e., the largest FPGA of the 65nm Radiation-Hardened-By-Design (RHBD) technology of NanoXplore to date. The proposed approach comprises a number of customized steps to systematically evaluate the entire FPGA design flow. Initially, we carefully select and tune a set of high-performance Digital Signal Processing (DSP) & Computer Vision (CV) benchmarks, which were originally developed as Hardware Description Language (HDL) IPs in past projects of the European Space Agency (ESA). Subsequently, we perform exhaustive exploration of the Synthesis, Placement, and Routing stages of the SW tools, as well as testing on actual HW boards. At each step, we generate and analyze a variety of results, while we also compare them to 3rd-party solutions. The results show that NG-Large provides sufficient programmability and performance, e.g., classic CV IPs for feature detection on megapixel images can achieve a throughput of 5–10 frames per second, while the on-chip memory utilization is up to 56% better than that of 3rd-party FPGAs. As a highlight, at system-level, we successfully implement and execute an entire HW/SW algorithmic pipeline for Vision-Based Navigation (VBN) involving SpaceWire data transfers with LEON CPU & NG-Large co-processing.

A Classification of Arab Ethnicity Based on Face Image Using Deep Learning Approach

Human face and facial features gain a lot of attention from researchers and are considered as one of the most popular topics recently. Features and information extracted from a person are known as soft biometric, they have been used to improve the recognition performance and enhance the search engine for face images, which can be further applied in various fields such as law enforcement, surveillance videos, advertisement, and social media profiling. By observing relevant studies in the field, we noted a lack of mention of the Arab world and an absence of Arab dataset as well. Therefore, our aim in this paper is to create an Arab dataset with proper labeling of Arab sub-ethnic groups, then classify these labels using deep learning approaches. Arab image dataset that was created consists of three labels: Gulf Cooperation Council countries (GCC), the Levant, and Egyptian. Two types of learning were used to solve the problem. The first type is supervised deep learning (classification); a Convolutional Neural Network (CNN) pre-trained model has been used as CNN models achieved state of art results in computer vision classification problems. The second type is unsupervised deep learning (deep clustering). The aim of using unsupervised learning is to explore the ability of such models in classifying ethnicities. To our knowledge, this is the first time deep clustering is used for ethnicity classification problems. For this, three methods were chosen. The best result of training a pre-trained CNN on the full Arab dataset then evaluating on a different dataset was 56.97%, and 52.12% when Arab dataset labels were balanced. The methods of deep clustering were applied on different datasets, showed an ACC from 32% to 59%, and NMI and ARI result from zero to 0.2714 and 0.2543 respectively.

Comparison of Classical Computer Vision vs. Convolutional Neural Networks for Weed Mapping in Aerial Images

In this paper, we present a comparison between convolutional neural networks and classicalcomputer vision approaches, for the specific precision agriculture problem of weed mapping on sugarcane fields aerial images. A systematic literature review was conducted to find which computer vision methods are being used on this specific problem. The most cited methods were implemented, as well as four models of convolutional neural networks. All implemented approaches were tested using the same dataset, and their results were quantitatively and qualitatively analyzed. The obtained results were compared to a human expert made ground truth, for validation. The results indicate that the convolutional neural networks present better precision and generalize better than the classical models

Using Cell Phone Pictures of Sheet Music To Retrieve MIDI Passages

This article investigates a cross-modal retrieval problem in which a user would like to retrieve a passage of music from a MIDI file by taking a cell phone picture of several lines of sheet music. This problem is challenging for two reasons: it has a significant runtime constraint since it is a user-facing application, and there is very little relevant training data containing cell phone images of sheet music. To solve this problem, we introduce a novel feature representation called a bootleg score which encodes the position of noteheads relative to staff lines in sheet music. The MIDI representation can be converted into a bootleg score using deterministic rules of Western musical notation, and the sheet music image can be converted into a bootleg score using classical computer vision techniques for detecting simple geometrical shapes. Once the MIDI and cell phone image have been converted into bootleg scores, we can estimate the alignment using dynamic programming. The most notable characteristic of our system is that it has no trainable weights at all -- only a set of about 40 hyperparameters. With a training set of just 400 images, we show that our system generalizes well to a much larger set of 1600 test images from 160 unseen musical scores. Our system achieves a test F measure score of 0.89, has an average runtime of 0.90 seconds, and outperforms baseline systems based on music object detection and sheet-audio alignment. We provide extensive experimental validation and analysis of our system.

Rethinking glottal midline detection

A healthy voice is crucial for verbal communication and hence in daily as well as professional life. The basis for a healthy voice are the sound producing vocal folds in the larynx. A hallmark of healthy vocal fold oscillation is the symmetric motion of the left and right vocal fold. Clinically, videoendoscopy is applied to assess the symmetry of the oscillation and evaluated subjectively. High-speed videoendoscopy, an emerging method that allows quantification of the vocal fold oscillation, is more commonly employed in research due to the amount of data and the complex, semi-automatic analysis. In this study, we provide a comprehensive evaluation of methods that detect fully automatically the glottal midline. We used a biophysical model to simulate different vocal fold oscillations, extended the openly available BAGLS dataset using manual annotations, utilized both, simulations and annotated endoscopic images, to train deep neural networks at different stages of the analysis workflow, and compared these to established computer vision algorithms. We found that classical computer vision perform well on detecting the glottal midline in glottis segmentation data, but are outperformed by deep neural networks on this task. We further suggest GlottisNet, a multi-task neural architecture featuring the simultaneous prediction of both, the opening between the vocal folds and the symmetry axis, leading to a huge step forward towards clinical applicability of quantitative, deep learning-assisted laryngeal endoscopy, by fully automating segmentation and midline detection.

Automatic Detection of Freshwater Phytoplankton Specimens in Conventional Microscopy Images.

Water safety and quality can be compromised by the proliferation of toxin-producing phytoplankton species, requiring continuous monitoring of water sources. This analysis involves the identification and counting of these species which requires broad experience and knowledge. The automatization of these tasks is highly desirable as it would release the experts from tedious work, eliminate subjective factors, and improve repeatability. Thus, in this preliminary work, we propose to advance towards an automatic methodology for phytoplankton analysis in digital images of water samples acquired using regular microscopes. In particular, we propose a novel and fully automatic method to detect and segment the existent phytoplankton specimens in these images using classical computer vision algorithms. The proposed method is able to correctly detect sparse colonies as single phytoplankton candidates, thanks to a novel fusion algorithm, and is able to differentiate phytoplankton specimens from other image objects in the microscope samples (like minerals, bubbles or detritus) using a machine learning based approach that exploits texture and colour features. Our preliminary experiments demonstrate that the proposed method provides satisfactory and accurate results.

Tomato Fruit Detection and Counting in Greenhouses Using Deep Learning.

Accurately detecting and counting fruits during plant growth using imaging and computer vision is of importance not only from the point of view of reducing labor intensive manual measurements of phenotypic information, but also because it is a critical step toward automating processes such as harvesting. Deep learning based methods have emerged as the state-of-the-art techniques in many problems in image segmentation and classification, and have a lot of promise in challenging domains such as agriculture, where they can deal with the large variability in data better than classical computer vision methods. This paper reports results on the detection of tomatoes in images taken in a greenhouse, using the MaskRCNN algorithm, which detects objects and also the pixels corresponding to each object. Our experimental results on the detection of tomatoes from images taken in greenhouses using a RealSense camera are comparable to or better than the metrics reported by earlier work, even though those were obtained in laboratory conditions or using higher resolution images. Our results also show that MaskRCNN can implicitly learn object depth, which is necessary for background elimination.

Welding defect detection: coping with artifacts in the production line

Visual quality inspection for defect detection is one of the main processes in modern industrial production facilities. In the last decades, artificial intelligence solutions took the place of classic computer vision techniques in the production lines and specifically in tasks that, for their complexity, were usually demanded to human workers yet obtaining similar or greater performance of their counterparts. This work exploits a Deep Neural Network for a smart monitoring system capable of performing accurate quality checks to detect welding defects in fuel injectors during the production stage. The contribution focuses on a novel approach to cope with unforeseen changes in production quality introduced by the alteration of a particular machine or process. Results suggest that pre-filtering could avoid the retraining of custom-designed networks. Moreover, the introduction of a weighting strategy on the confusion matrix allows obtaining good performance estimations even in the case of small and unbalanced datasets. Concerning a specific demanding case of an imbalanced dataset with very few positive examples, the system displayed a 96.30% accuracy on defect classification.

A distributed cognitive approach in cybernetic modelling of human vision in a robotic swarm

Abstract Objectives In this paper a novel approach regarding image analysis in Machine Vision applications was proposed. Methods The presented concept consists of two issues: (1) shifting some of the complex image processing and understanding algorithms from a mobile robot to distributed computer, and (2) designing the cognitive system (in a distributed computer) in such a way, that it would be common for numerous robots. The authors of this work focused on image processing, and they propose to accelerate vision understanding by using Cooperative Vision (CoV), i.e., to get video input from cooperating robots and process it in a centralized system. Results To verify the purposefulness of such approach, a comparative study is currently being conducted, involving a classical single-camera Computer Vision (CV) mobile robot and two (or more) single-camera CV robots cooperating in CoV mode. Conclusions The CoV system is being designed and implemented so that the algorithm would be able to utilize multiple video sources for recognition of objects on the very same scene.

Deep Learning-Based Water Segmentation for Autonomous Surface Vessel

Visual-based obstacle detection from an autonomous surface vessel (ASV) is a complex task due to high variance of scene properties such as different illumination and presence of reflections. One approach in implementing the task is through extracting waterlines to enable inferring of vessel orientation and obstacles presence. Classical computer vision algorithms for detection holds limitation in robustness and scalability. With recent breakthroughs in deep neural network architectures, vision-based object detection is seen to obtain high performance. In this work, the deep learning models based on Convolutional Neural Network (CNN) to implement binary semantic segmentation is studied. This architecture identifies each pixel to water and non-water classes. In purpose of benchmarking models, Fully Convolutional Network (FCN), SegNet and U-Net are trained on a publicly available dataset, IntCatch Vision Data Set (ICVDS), to evaluate the performance. From the experiments carried out, quantitative results show effectiveness of the models with accuracy all above 95.55% and lowest average speed of 11 frames per second. To improve, pre-trained networks (VGG 16, Resnet-50 and MobileNet) are used as a backbone, obtaining an improved accuracy above 98.14% with lowest inferring speed of 10 frame per second. Using the developed ASV, new dataset of 143 images called Malaysia ASV Dataset (MASVD) is collected, labelled and made publicly available. The trained models are tested with the newly collected dataset obtaining accuracy of 75%. The high accuracy performance shows potential for the models to be employed for collision avoidance algorithm in ASV navigation.

DEVELOPING A COMPUTER VISION RE-IDENTIFICATION SYSTEM

Background. The rapid growth of computational power of machines and amount of data caused exploding usage of computer vision in a large variety of tasks and in particular for people recognition.Objective. The aim of the paper is to propose a computer vision re-identification system based on research. Also improvements for detection and recognition models of the system are made.Methods. We used classical computer vision and deep learning techniques to create the system.Results. The main contribution of the research is a description of the optimal system structure with a trade-of between speed and quality. Furthermore, requirements for an environment are proposed, which allows to set up the system in the real world with guaranteed quality.Conclusions. Real-time computer vision re-identification system was developed and can be used in a production environment which satisfy requirements.

A Material-Sensing Time-of-Flight Camera

Time-of-flight (ToF) cameras are active imaging sensors able to capture the 3-D geometry of a scene. Objects can be then segmented and recognized from their 3-D geometry. Only very recently it has been shown that ToF cameras can also recognize the material on a per-pixel basis, as opposed to classical computer vision approaches, which rely on texture and operate per patches or regions. Differently, ToF-based material recognition takes profit of differences between the temporal impulse response function of different materials to distinguish them. Existing approaches use either raw data or processed depth measurements at different modulation frequencies to generate distinctive feature vectors for classification. In this letter we stress the importance of the bandlimited character of the material impulse response function (MIRF) and propose direct Fourier sampling. These Fourier coefficients are then used as set of features after appropriate complex scaling. Using orthogonal sensing functions enables classification with fewer but more informative measurements. Minimizing the number of measurements speeds up both acquisition, training, and classification. Moreover, 2-D superpixel segmentation in feature space allows for reliable classification with few queries per region. We demonstrate that the proposed approach allows for real-time simultaneous depth- and material-sensing ToF imaging.