Visual Recognition System Research Articles

Towards seamless human robot collaboration: integrating multimodal interaction

This paper discusses the challenges in the collaboration between human operators and industrial robots for assembly operations focusing on safety and simplified interaction. A case study is presented, involving perception technologies for the robot in conjunction with wearable devices used by the operator. In terms of robot perception, a manual guidance module, an air pressor contact sensor namely skin, and a vision system for recognition and tracking of objects have been developed and integrated. Concerning the wearable devices, an advanced user interface including audio and haptic commands accompanied by augmented reality technology are used to support the operator and provide awareness by visualizing information related to production and safety aspects. In parallel, safety functionalities are implemented through collision detection technologies such as a safety skin and safety monitored regions delimiting the area of the robot activities. The complete system is coordinated under a common integration platform and it is validated in a case study of the white goods industry.

An Algorithm to Identify Syllable from a Visual Speech Recognition System

This paper is to propose a highly efficient and reliable real time communication system for speech impaired people to communicate and converse in an effective manner. The main idea deals with an algorithm to identify the word from a visual speech input disregard with its acoustic property. The non-acoustic speech is captured through a source and given as an input in the form of image frames then classified to obtain the desired form of output. The input given in the visual form deals with mouth postures. The network is structured to identify the speech in form of syllables. Convolution Neural Network, a deep learning technique is used as its classifier. A database is created especially for this algorithm and are aligned within in the form of class and subsets.

Brain-Like Navigation Scheme based on MEMS-INS and Place Recognition

Animals have certain cognitive competence about the environment so they can correct their navigation errors. Inspired by the excellent navigational behavior of animals, this paper proposes a brain-like navigation scheme to improve the accuracy and intelligence of Micro-Electro-Mechanical System based Inertial Navigation Systems (MEMS-INS). The proposed scheme employs vision to acquire external perception information as an absolute reference to correct the position errors of INS, which is established by analyzing the navigation and error correction mechanism of rat brains. In addition, to improve the place matching speed and precision of the system for visual scene recognition, this paper presents a novel place recognition algorithm that combines image scanline intensity (SI) and grid-based motion statistics (GMS) together which is named the SI-GMS algorithm. The proposed SI-GMS algorithm can effectively reduce the influence of uncertain environment factors on the recognition results, such as pedestrians and vehicles. It solves the problem that the matching result will occasionally go wrong when simply using the scanline intensity (SI) algorithm, or the slow matching speed when simply using grid-based motion statistics (GMS) algorithm. Finally, an outdoor Unmanned Aerial Vehicle (UAV) flight test is carried out. Based on the reference information from the high-precision GPS device, the results illustrate the effectiveness of the scheme in error correction of INS and the algorithm in place recognition.

Measuring the effect of high-speed video data on the audio-visual speech recognition accuracy

Introduction: The effectiveness of modern automatic speech recognition systems in quiet acoustic conditions is quite high and reaches 90-95%. However, in noisy uncontrolled environment, acoustic signals are often distorted, which greatly reduces the resulting recognition accuracy. In adverse conditions, it seems appropriate to use the visual information about the speech, as it is not affected by the acoustic noise. At the moment, there are no studies which objectively reflect the dependence of visual speech recognition accuracy on the video frame rate, and there are no relevant audio-visual databases for model training. Purpose : Improving the reliability and accuracy of the automatic audio-visual Russian speech recognition system; collecting representative audio-visual database and developing an experimental setup. Methods : For audio-visual speech recognition, we used coupled hidden Markov model architectures. For parametric representation of audio and visual features, we used mel-frequency cepstral coefficients and principal component analysis-based pixel features. Results: In the experiments, we studied 5 different rates of video data: 25, 50, 100, 150, and 200 fps. Experiments have shown a positive effect from the use of a high-speed video camera: we achieved an absolute increase in accuracy of 1.48% for a bimodal system and 3.10% for a unimodal one, as compared to the standard recording speed of 25 fps. During the experiments, test data for all speakers were added with two types of noise: wide-band white noise and “babble noise”. Analysis shows that bimodal speech recognition exceeds unimodal in accuracy, especially for low SNR values <15 dB. At very low SNR values <5 dB, the acoustic information becomes non-informative, and the best results are achieved by a unimodal visual speech recognition system. Practical relevance : The use of a high-speed camera can improve the accuracy and robustness of a continuous audio-visual Russian speech recognition system.

Multi-Process Fusion: Visual Place Recognition Using Multiple Image Processing Methods

Typical attempts to improve the capability of visual place recognition techniques include the use of multi-sensor fusion and integration of information over time from image sequences. These approaches can improve performance but have disadvantages including the need for multiple physical sensors and calibration processes, both for multiple sensors and for tuning the image matching sequence length. In this paper we address these shortcomings with a novel "multi-sensor" fusion approach applied to multiple image processing methods for a single visual image stream, combined with a dynamic sequence matching length technique and an automatic processing method weighting scheme. In contrast to conventional single method approaches, our approach reduces the performance requirements of a single image processing methodology, instead requiring that within the suite of image processing methods, at least one performs well in any particular environment. In comparison to static sequence length techniques, the dynamic sequence matching technique enables reduced localization latencies through analysis of recognition quality metrics when re-entering familiar locations. We evaluate our approach on multiple challenging benchmark datasets, achieving superior performance to two state-of-the-art visual place recognition systems across environmental changes including winter to summer, afternoon to morning and night to day. Across the four benchmark datasets our proposed approach achieves an average F1 score of 0.96, compared to 0.78 for NetVLAD and 0.49 for SeqSLAM. We provide source code for the multi-fusion method and present analysis explaining how superior performance is achieved despite the multiple, disparate, image processing methods all being applied to a single source of imagery, rather than to multiple separate sensors.

Analysis of Customer Emotion from Video based Feedback of a Product

Due to the high levels of competition in a global market, companies have put more effort on building strong customer relationships and increasing customer satisfaction levels. Now-a-days due to technological improvements in information and communication technologies gives a highly anticipated key contributor to improve the customer experience and satisfaction in service episodes is through the application of video analytics, such as to evaluate the customer’s emotions over the complete service cycle. Currently, emotion recognition from video could be a difficult analysis space. One of the foremost effective solutions to deal with this challenge is to utilize each audio and visual part as two sources contained within the video knowledge to form an overall assessment of the emotion. The combined use of audio and visual knowledge sources presents further challenges, such as determining the optimal data fusion technique prior to classification. In this paper, we propose an audio–visual emotion recognition system to detect the universal six emotions (happy, angry, sad, disgust, surprise, and fear) from video data. The detected customer emotions are then mapped and translated to provide client satisfaction scores. The projected client satisfaction video analytics system will operate over video conferencing or video chat. The effectiveness of our proposal is verified through numerical results.

Revisiting the Bag-of-Visual-Words model: A hierarchical localization architecture for mobile systems

In this paper, an enhanced visual place recognition system is proposed aiming to improve the localization performance of a mobile platform. Our technique takes full advantage of the continuous input image stream in order to provide additional knowledge to the matching functionality. The well-established Bag-of-Visual-Words model is adapted into a hierarchical design that derives the visual information from the full entity of a natural scene into the description, while it additionally preserves the geometric structure of the explored world. Our approach is evaluated as part of a state-of-the-art Simultaneous-Localization-and-Mapping algorithm, and parallelization techniques are exploited utilizing every available hardware module in a low-power device. The implemented algorithm has been tested on several publicly available datasets offering consistently accurate localization results and preventing the majority of redundant computations that the additional geometrical verifications can induce.

Visual Object Recognition and Pose Estimation Based on a Deep Semantic Segmentation Network

In recent years, deep learning-based object recognition algorithms become emerging in robotic vision applications. This paper addresses the design of a novel deep learning-based visual object recognition and pose estimation system for a robot manipulator to handle random object picking tasks. The proposed visual control system consists of a visual perception module, an object pose estimation module, a data argumentation module, and a robot manipulator controller. The visual perception module combines deep convolution neural networks (CNNs) and a fully connected conditional random field layer to realize an image semantic segmentation function, which can provide stable and accurate object classification results in cluttered environments. The object pose estimation module implements a model-based pose estimation method to estimate the 3D pose of the target for picking control. In addition, the proposed data argumentation module automatically generates training data for training the deep CNN. Experimental results show that the proposed scene segmentation method used in the data argumentation module reaches a high accuracy rate of 97.10% on average, which is higher than other state-of-the-art segment methods. Moreover, with the proposed data argumentation module, the visual perception module reaches an accuracy rate over than 80% and 72% in the case of detecting and recognizing one object and three objects, respectively. In addition, the proposed model-based pose estimation method provides accurate 3D pose estimation results. The average translation and rotation errors in the three axes are all smaller than 0.52 cm and 3.95 degrees, respectively. These advantages make the proposed visual control system suitable for applications of random object picking and manipulation.

Oocyte Positional Recognition for Automatic Manipulation in ICSI.

Polar body position detection is a necessary process in the automation of micromanipulation systems specifically used in intracytoplasmic sperm injection (ICSI) applications. The polar body is an intracellular structure, which accommodates the chromosomes, and the injection must not only avoid this structure but be at the furthest point away from it. This paper aims to develop a vision recognition system for the recognition of the oocyte and its polar body in order to be used to inform the automated injection mechanism to avoid the polar body. The novelty of the paper is its capability to determine the position and orientation of the oocyte and its polar body. The gradient-weighted Hough transform method was employed for the detection of the location of the oocyte and its polar body. Moreover, a new elliptical fitting method was employed for size measurement of the polar bodies and oocytes for the allowance of morphological variance of the oocytes and their polar bodies. The proposed algorithm has been designed to be adaptable with typical commercial inverted microscopes with different criteria. The successful experimental results for this algorithm produce maximum errors of 5% for detection and 10% for reporting respectively.

An adaptive localization system for image storage and localization latency requirements

Fast and efficient global localization is a critical problem for autonomous systems. Existing sequence-based visual place recognition requires a storage-intensive image database for robust localization, while more storage-efficient odometry-based place recognition approaches can require a long travel distance to obtain an accurate localization. In this paper, we present a novel particle filter-based localization system that adapts to varying degrees of map image densities, road layout ambiguity and visual appearance change. The base system combines a geometric place recognition capability utilizing odometry and roadmaps with a visual place recognition system. When using a sparse image database, particles could exist at visually unknown places, which introduces difficulties in performing sequential visual place recognition. To address this challenge, we propose to make use of effective visual observations to enable the system to accumulate visual belief sequentially, even when reference images are very sparse. Furthermore, we develop a vision reliability estimation method, which analyses the relationship between the visual component and the particle filter convergence, to calibrate the optimal contribution of vision to particle weighting in different visual environments and conditions. To evaluate our approach, we perform extensive experiments using four benchmark localization datasets, and control the reference image density by subsampling these datasets. Results show that the proposed technique is able to consistently and correctly localize the vehicle over a range of reference image densities, and to consistently outperform a particle filter-enhanced version of an existing state-of-the-art SeqSLAM system, which fails when image spacing exceeds 30 m. In particular, for a 600% increase in database image sparsity (from 10 m to 70 m), we show that the proposed method is able to maintain localization performance with only a 40% increase in localization latency (from 250 m to 350 m). We also provide an analysis of the results and a characterization of the system’s computational requirements.

A Retinotopic Spiking Neural Network System for Accurate Recognition of Moving Objects Using NeuCube and Dynamic Vision Sensors.

This paper introduces a new system for dynamic visual recognition that combines bio-inspired hardware with a brain-like spiking neural network. The system is designed to take data from a dynamic vision sensor (DVS) that simulates the functioning of the human retina by producing an address event output (spike trains) based on the movement of objects. The system then convolutes the spike trains and feeds them into a brain-like spiking neural network, called NeuCube, which is organized in a three-dimensional manner, representing the organization of the primary visual cortex. Spatio-temporal patterns of the data are learned during a deep unsupervised learning stage, using spike-timing-dependent plasticity. In a second stage, supervised learning is performed to train the network for classification tasks. The convolution algorithm and the mapping into the network mimic the function of retinal ganglion cells and the retinotopic organization of the visual cortex. The NeuCube architecture can be used to visualize the deep connectivity inside the network before, during, and after training and thereby allows for a better understanding of the learning processes. The method was tested on the benchmark MNIST-DVS dataset and achieved a classification accuracy of 92.90%. The paper discusses advantages and limitations of the new method and concludes that it is worth exploring further on different datasets, aiming for advances in dynamic computer vision and multimodal systems that integrate visual, aural, tactile, and other kinds of information in a biologically plausible way.

End-to-End Policy Learning for Active Visual Categorization.

Visual recognition systems mounted on autonomous moving agents face the challenge of unconstrained data, but simultaneously have the opportunity to improve their performance by moving to acquire new views at test time. In this work, we first show how a recurrent neural network-based system may be trained to perform end-to-end learning of motion policies suited for this "active recognition" setting. Further, we hypothesize that active vision requires an agent to have the capacity to reason about the effects of its motions on its view of the world. To verify this hypothesis, we attempt to induce this capacity in our active recognition pipeline, by simultaneously learning to forecast the effects of the agent's motions on its internal representation of the environment conditional on all past views. Results across three challenging datasets confirm both that our end-to-end system successfully learns meaningful policies for active category recognition, and that "learning to look ahead" further boosts recognition performance.

Demonstration of Unsupervised Learning With Spike-Timing-Dependent Plasticity Using a TFT-Type NOR Flash Memory Array

We investigate the characteristics of a synaptic imitation device using a thin-film transistor (TFT)-type NOR flash memory cell with a half-covered floating gate. The long-term potentiation (LTP) and long-term depression (LTD) required for the operation of the spike-timing-dependent plasticity (STDP) algorithm are implemented using the proposed pulse scheme. Unsupervised learning is successfully demonstrated by applying the STDP learning rule through software MATLAB simulation reflecting the LTP/LTD characteristics of the fabricated TFT-type NOR flash memory array. We present the learning and recognition processes of $28\times28$ MNIST handwritten digit patterns. First, STDP learning in a single-neuron string ( $784\times1$ ) is investigated, after which STDP learning is demonstrated in a multineuron array ( $784\times10$ ) with a lateral inhibition function to demonstrate the ability of multipattern learning and recognition. Meanwhile, we investigate the key factors of STDP unsupervised learning. Finally, an approach is suggested to implement a hardware neural network using the conventional CMOS technology for STDP unsupervised learning as a visual pattern recognition system.

The impact of the Lombard effect on audio and visual speech recognition systems

When producing speech in noisy backgrounds talkers reflexively adapt their speaking style in ways that increase speech-in-noise intelligibility. This adaptation, known as the Lombard effect, is likely to have an adverse effect on the performance of automatic speech recognition systems that have not been designed to anticipate it. However, previous studies of this impact have used very small amounts of data and recognition systems that lack modern adaptation strategies. This paper aims to rectify this by using a new audio-visual Lombard corpus containing speech from 54 different speakers – significantly larger than any previously available – and modern state-of-the-art speech recognition techniques.The paper is organised as three speech-in-noise recognition studies. The first examines the case in which a system is presented with Lombard speech having been exclusively trained on normal speech. It was found that the Lombard mismatch caused a significant decrease in performance even if the level of the Lombard speech was normalised to match the level of normal speech. However, the size of the mismatch was highly speaker-dependent thus explaining conflicting results presented in previous smaller studies. The second study compares systems trained in matched conditions (i.e., training and testing with the same speaking style). Here the Lombard speech affords a large increase in recognition performance. Part of this is due to the greater energy leading to a reduction in noise masking, but performance improvements persist even after the effect of signal-to-noise level difference is compensated. An analysis across speakers shows that the Lombard speech energy is spectro-temporally distributed in a way that reduces energetic masking, and this reduction in masking is associated with an increase in recognition performance. The final study repeats the first two using a recognition system training on visual speech. In the visual domain, performance differences are not confounded by differences in noise masking. It was found that in matched-conditions Lombard speech supports better recognition performance than normal speech. The benefit was consistently present across all speakers but to a varying degree. Surprisingly, the Lombard benefit was observed to a small degree even when training on mismatched non-Lombard visual speech, i.e., the increased clarity of the Lombard speech outweighed the impact of the mismatch.The paper presents two generally applicable conclusions: i) systems that are designed to operate in noise will benefit from being trained on well-matched Lombard speech data, ii) the results of speech recognition evaluations that employ artificial speech and noise mixing need to be treated with caution: they are overly-optimistic to the extent that they ignore a significant source of mismatch but at the same time overly-pessimistic in that they do not anticipate the potential increased intelligibility of the Lombard speaking style.

FCN and LSTM Based Computer Vision System for Recognition of Vehicle Type, License Plate Number, and Registration Country

We propose an advanced Automatic number-plate recognition (ANPR) system, which not only recognizes the number and the issuing state, but also the type and location of the vehicle in the input image. The system is based on a combination of existing methods, modifications to neural network architectures and improvements in the training process. The proposed system uses machine-learning approach and consists of three main parts: segmentation of input image by Fully Convolutional Network for localization of license plate and determination of vehicle type; recognition of the characters of the localized plate by a Maxout CNN and LSTM; determination of the state that has issued the license plate by a CNN. The training of these neural network models is accomplished using a manually labeled custom dataset, which is expanded with data augmented techniques. The resulting system is capable of localizing and classifying multiple types of vehicles (including motorcycles and emergency vehicles) as well as their license plates. The achieved precision of the localization is 99.5%. The whole number recognition accuracy is 96.7% and character level recognition accuracy is 98.8%. The determination of issuing state is precise in 92.8% cases.

A Visual Coding Method for Geographic Statistics Based on the Pattern Recognition Feature of Optic Nerve in Cerebral Cortex

The human visual system can easily identify a variety of objects, all thanks to its powerful pattern recognition capability. One theory holds that the brain’s visual recognition mechanism is mainly achieved by single neurons and complex neurons located in area V1 of the cerebral cortex. Both types of cells decompose and synthesize the visual signals from sensory organs to extract their pattern features (Riesnhuber et al., 1991). However, in the information visualization field where logic is quite complicated, the visual recognition system of human beings has great limitations and can only effectively recognize complex visual patterns after the complex information is pretreated by a set of scientific visual coding methods. In the context of geographic statistics, based on the single neurons and complex neurons model and Gestalt psychology, this paper proposes a visual coding method based on aggregation and subdivision (AS method) to visualize geographic statistics. The simulation test results show that the AS method can deliver a good mapping relationship between geographic locations and a good rectangular aspect ratio and thus can achieve high visual perception efficiency.

Application of Deep Learning for Object Detection

The ubiquitous and wide applications like scene understanding, video surveillance, robotics, and self-driving systems triggered vast research in the domain of computer vision in the most recent decade. Being the core of all these applications, visual recognition systems which encompasses image classification, localization and detection have achieved great research momentum. Due to significant development in neural networks especially deep learning, these visual recognition systems have attained remarkable performance. Object detection is one of these domains witnessing great success in computer vision. This paper demystifies the role of deep learning techniques based on convolutional neural network for object detection. Deep learning frameworks and services available for object detection are also enunciated. Deep learning techniques for state-of-the-art object detection systems are assessed in this paper.

Application of an artificial immune system for visual pattern recognition

The suitability of artificial immune systems for recognizing visual patterns is discussed. A new algorithm and software implementation of an artificial immune system have been proposed based on which real-time pattern recognition can be done using a Web camera. It has been shown experimentally that this system can be successfully used to recognize both human faces and any other objects. An issue of using an artificial immune system in high-performance parallel computing systems is discussed. The advantages of the developed artificial immune system include the ability to teach the system a new image in a fast manner at any moment during run-time. These advantages open up a possibility of creating artificial intelligence systems for real-time learning.

Visual recognition system of cherry picking robot based on Lab color model

This paper designs a visual recognition system suitable for cherry picking. First, the system deals with the image using the vector median filter. And then it extracts a channel of Lab color model to divide the cherries and the background. The cherry contour was successfully fitted by the least square method, and the centroid and radius of the cherry were extracted. Finally, the cherry was successfully extracted.

Evaluation of Object Proposals and ConvNet Features for Landmark-based Visual Place Recognition

Despite significant progress has been made in visual place recognition for mobile robot navigation, challenges remain, especially in changing environments. Recently, a landmark-based visual place description technique has achieved impressive results under conditions of significant environmental and viewpoint changes, raising the interest of the community in it. This technique combines the strengths of object proposals and convolutional neural networks (ConvNets), which are the latest achievements in object detection and deep learning research. The idea is to detect landmarks in an image with an object proposal method and then characterize these landmarks as features (known as ConvNet features) computed by a ConvNet for matching landmarks. Although a large number of object proposal approaches and ConvNet features have been proposed, it remains unclear how to select or combine object proposals and ConvNet features for a landmark-based visual place recognition system. In this paper we conduct a thorough evaluation of 13 state-of-the-art object proposal methods and 13 kinds of modern ConvNet features on six datasets with various environmental and viewpoint changes, in terms of their place recognition accuracy and computational efficiency. Our study identifies the strengths and weaknesses of object proposal methods and ConvNet features with respect to environmental changes. Conclusions drawn from our analysis are expected to be useful for developing landmark-based visual place recognition systems and benefit other related research fields.