Embedded Computer Vision Research Articles

Mobile Robot Navigation Based on Embedded Computer Vision

The current computational advance allows the development of technological solutions using tools, such as mobile robots and programmable electronic systems. We present a design that integrates the Khepera IV mobile robot with an NVIDIA Jetson Xavier NX board. This system executes an algorithm for navigation control based on computer vision and the use of a model for object detection. Among the functionalities that this integration adds to the Khepera IV in generating guided driving are trajectory tracking for safe navigation and the detection of traffic signs for decision-making. We built a robotic platform to test the system in real time. We also compared it with a digital model of the Khepera IV in the CoppeliaSim simulator. The navigation control results show significant improvements over previous works. This is evident in both the maximum navigation speed and the hit rate of the traffic sign detection system. We also analyzed the navigation control, which achieved an average success rate of 93%. The architecture allows testing new control techniques or algorithms based on Python, facilitating future improvements.

Intelligent potato frying: Time to say goodbye to the “good old” processing strategies

• The appearance of intelligent potato friers is imminent. • Computer vision is promising in quality inspection of raw and processed potatoes. • New emerging pre-treatment technologies increase quality of potato chips. • Dynamic optimization is superior to static optimization in potato frying process. • Fully fledged Digital Twins will facilitate predictive control of product quality. Potato chips production is a traditional food process. To achieve uniform product quality, raw materials are usually rigorously sorted. Traditionally, the process is conducted in a single stage approach leading to high quality losses. Recently, dynamically optimized frying processes have been found to result in higher product quality. Consequently, industrial continuous deep-fat fryers convey potato disks through several zones pre-set at different temperatures. However, these improved systems still do not take the variabilities in frying kinetics among potatoes into consideration. To address this issue and decrease uncertainties in end-product quality, frying conditions of each zone must be optimized, physiochemical properties of the various raw tubers and their frying kinetics taking into account. This paper, therefore, presents a novel approach for an intelligent frying process with embedded computer vision systems providing continuous monitoring of product quality and, therefore, facilitate dynamic control of frying conditions in order to meet desired quality attributes in the final product. An extensive literature review of the key physiochemical attributes of raw potato tubers is presented, followed by an introduction to novel pre-treatment technologies, and the importance of optimal frying conditions. An overview of the potentials for using computer vision systems for the assessment of said quality criteria is given, followed by a detailed description of the envisioned frying process. The paper concludes that the realization of intelligent frying processes necessitates the development of fully fledged digital twins of the process and the products, combining physics based and data driven modelling with real time sensing and control. Terminology: Chips refer to thin slices of potato while French fries refers to wedges/stripes.

매니퓰레이터 로봇의 기구학 파라미터 보정에 사용되는 구조광의 정밀성 향상을 위한 영상처리 알고리즘

The machining tolerance and assembly error in the robot mechanism cause deterioration of the control precision of the articulated manipulator robot. To handle this problem, a correction algorithm should be used. The algorithm can be classified into a contact-based method and a non-contact-based method. This paper deals with a non-contact correction method using a structured light and subsequent image processing algorithm to estimate a laser point projected onto a screen to improve the measurement accuracy of non-contact method. In the conventional method, the laser image on the screen observed by the camera was simply assumed to be a single point. However, the actual observed image has an area of a size that is difficult to ignore. In addition, there is a significant time delay between the actual point and the captured point from the frame grabber of the computer vision. Consequently, these reduce the estimation accuracy. To improve the estimation performance, mathematical model-based image processing algorithm is proposed to handle these two major prerequisites. Testbed system was developed with an embedded computer vision system to implement the algorithm. The performance was verified by comparing the actual point from the high resolution encoder signal and the estimated point from the proposed algorithm.

Deep Learning Method for Recognition and Classification of Images from Video Recorders in Difficult Weather Conditions

The sustainable functioning of the transport system requires solving the problems of identifying and classifying road users in order to predict the likelihood of accidents and prevent abnormal or emergency situations. The emergence of unmanned vehicles on urban highways significantly increases the risks of such events. To improve road safety, intelligent transport systems, embedded computer vision systems, video surveillance systems, and photo radar systems are used. The main problem is the recognition and classification of objects and critical events in difficult weather conditions. For example, water drops, snow, dust, and dirt on camera lenses make images less accurate in object identification, license plate recognition, vehicle trajectory detection, etc. Part of the image is overlapped, distorted, or blurred. The article proposes a way to improve the accuracy of object identification by using the Canny operator to exclude the damaged areas of the image from consideration by capturing the clear parts of objects and ignoring the blurry ones. Only those parts of the image where this operator has detected the boundaries of the objects are subjected to further processing. To classify images by the remaining whole parts, we propose using a combined approach that includes the histogram-oriented gradient (HOG) method, a bag-of-visual-words (BoVW), and a back propagation neural network (BPNN). For the binary classification of the images of the damaged objects, this method showed a significant advantage over the classical method of convolutional neural networks (CNNs) (79 and 65% accuracies, respectively). The article also presents the results of a multiclass classification of the recognition objects on the basis of the damaged images, with an accuracy spread of 71 to 86%.

A Hybrid 3D Printed Hand Prosthesis Prototype Based on sEMG and a Fully Embedded Computer Vision System.

This study presents a new approach for an sEMG hand prosthesis based on a 3D printed model with a fully embedded computer vision (CV) system in a hybrid version. A modified 5-layer Smaller Visual Geometry Group (VGG) convolutional neural network (CNN), running on a Raspberry Pi 3 microcomputer connected to a webcam, recognizes the shape of daily use objects, and defines the pattern of the prosthetic grasp/gesture among five classes: Palmar Neutral, Palmar Pronated, Tripod Pinch, Key Grasp, and Index Finger Extension. Using the Myoware board and a finite state machine, the user's intention, depicted by a myoelectric signal, starts the process, photographing the object, proceeding to the grasp/gesture classification, and commands the prosthetic motors to execute the movements. Keras software was used as an application programming interface and TensorFlow as numerical computing software. The proposed system obtained 99% accuracy, 97% sensitivity, and 99% specificity, showing that the CV system is a promising technology to assist the definition of the grasp pattern in prosthetic devices.

Detecting Diabetic Retinopathy Using Embedded Computer Vision

Diabetic retinopathy is one of the leading causes of vision loss in the United States and other countries around the world. People who have diabetic retinopathy may not have symptoms until the condition becomes severe, which may eventually lead to vision loss. Thus, the medically underserved populations are at an increased risk of diabetic retinopathy-related blindness. In this paper, we present development efforts on an embedded vision algorithm that can classify healthy versus diabetic retinopathic images. Convolution neural network and a k-fold cross-validation process were used. We used 88,000 labeled high-resolution retina images obtained from the publicly available Kaggle/EyePacs database. The trained algorithm was able to detect diabetic retinopathy with up to 76% accuracy. Although the accuracy needs to be further improved, the presented results represent a significant step forward in the direction of detecting diabetic retinopathy using embedded computer vision. This technology has the potential of being able to detect diabetic retinopathy without having to see an eye specialist in remote and medically underserved locations, which can have significant implications in reducing diabetes-related vision losses.

An Embedded Computer Vision System for Beans Quality Inspection

An Embedded Computer Vision System for Beans Quality Inspection

A Low-Cost High-Speed Object Tracking VLSI System Based on Unified Textural and Dynamic Compressive Features

Real-time object tracking is demanded in versatile embedded computer vision applications. To this end, a low-cost high-speed VLSI system is proposed for object tracking, based on robust and computationally simple unified textural and dynamic compressive sensing features and simple elliptic matching with fast online template updating capability. The system introduces a memory-centric architectural paradigm, multiple-level pipelines and parallel processing circuits to achieve high frame rate while consuming as few hardware resources as possible. We implemented an FPGA prototype of the proposed VLSI tracking system. Under a 100 MHz clock frequency, the prototype achieves over 600 frame/s processing speed under <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$320\times240$ </tex-math></inline-formula> image resolution and obtains robust tracking results.

PDCAT: a framework for fast, robust, and occlusion resilient fiducial marker tracking

Square binary patterns have become the de facto fiducial marker for most computer vision applications. Existing tracking solutions suffer a number of limitations, such as the low frame-rate and sensitivity to partial occlusions. This work aims at overcoming these limitations, by exploiting temporal information in video-sequences. We propose a parallel detection, compensation and tracking (PDCAT) framework, which can be integrated into any binary marker system. Our solution is capable of recovering markers even when they become mostly occluded. Furthermore, the low processing time of the tracking task makes PDCAT more than an order of magnitude faster than a track-by-detect solution. This is particularly important for embedded computer vision applications, wherein the detection run at a very low frame rate. In the experiments conducted on an embedded computer, the processing frame rate of the track-by-detect solution was merely 11 FPS. Our solution, on the other hand, was capable of processing more than 100 FPS.

A method for embedding a computer vision application into a wearable device

Pattern classification applications can be found everywhere, especially the ones that use computer vision. What makes them difficult to embed is the fact that they often require a lot of computational resources. Embedded computer vision has been applied in many contexts, such as industrial or home automation, robotics, and assistive technologies. This work performs a design space exploration in an image classification system and embeds a computer vision application into a minimum resource platform, targeting wearable devices. The feature extractor and the classifier are evaluated for memory usage and computation time. A method is proposed to optimize such characteristics, leading to a reduction of over 99% in computation time and 92% in memory usage, with respect to a standard implementation. Experimental results in an ARM Cortex-M platform showed a total classification time of 0.3 s, maintaining the same accuracy as in the simulation performed. Furthermore, less than 20 KB of data memory was required, which is the most limited resource available in low-cost and low-power microcontrollers. The target application, used for the experimental evaluation, is a crosswalk detector used to help visually impaired persons.

FPGA Based Optimized Reconfigurable Base-2 Constant Coefficient Multiplier Architecture for Image Filtering

Image convolution using FPGA has been comprehensively used for noise removal of Reconfigurable computing based image Processing Algorithm. Particularly these filters are widely used in embedded computer vision applications like edge detection and Feature extraction analysis. Practical implementation of filter requires enormous computational requirement. The multiplier plays very important role in the image convolution. The existed multiplier design requires more computational complexity for the 3x3 test image. For this the proposed reconfigurable constant coefficient multiplier uses base-2 Common sub expression algorithm. which reduces the computational complexity in a better way. The proposed 2D-convolution in image application is the value of resultant output is multiplication of image pixel with corresponding kernel value. In this work the realization of 2D convolution to be done using proposed constant coefficient multiplier and analyzed using Xilinx Virtex-5 FPGA platform

Smart Doll: Emotion Recognition Using Embedded Deep Learning

Computer vision and deep learning are clearly demonstrating a capability to create engaging cognitive applications and services. However, these applications have been mostly confined to powerful Graphic Processing Units (GPUs) or the cloud due to their demanding computational requirements. Cloud processing has obvious bandwidth, energy consumption and privacy issues. The Eyes of Things (EoT) is a powerful and versatile embedded computer vision platform which allows the user to develop artificial vision and deep learning applications that analyse images locally. In this article, we use the deep learning capabilities of an EoT device for a real-life facial informatics application: a doll capable of recognizing emotions, using deep learning techniques, and acting accordingly. The main impact and significance of the presented application is in showing that a toy can now do advanced processing locally, without the need of further computation in the cloud, thus reducing latency and removing most of the ethical issues involved. Finally, the performance of the convolutional neural network developed for that purpose is studied and a pilot was conducted on a panel of 12 children aged between four and ten years old to test the doll.

Guest Editorial: Special Issue on Embedded Computer Vision

Guest Editorial: Special Issue on Embedded Computer Vision

TOWARDS GUIDED UNDERWATER SURVEY USING LIGHT VISUAL ODOMETRY

Abstract. A light distributed visual odometry method adapted to embedded hardware platform is proposed. The aim is to guide underwater surveys in real time. We rely on image stream captured using portable stereo rig attached to the embedded system. Taken images are analyzed on the fly to assess image quality in terms of sharpness and lightness, so that immediate actions can be taken accordingly. Images are then transferred over the network to another processing unit to compute the odometry. Relying on a standard ego-motion estimation approach, we speed up points matching between image quadruplets using a low level points matching scheme relying on fast Harris operator and template matching that is invariant to illumination changes. We benefit from having the light source attached to the hardware platform to estimate a priori rough depth belief following light divergence over distance low. The rough depth is used to limit points correspondence search zone as it linearly depends on disparity. A stochastic relative bundle adjustment is applied to minimize re-projection errors. The evaluation of the proposed method demonstrates the gain in terms of computation time w.r.t. other approaches that use more sophisticated feature descriptors. The built system opens promising areas for further development and integration of embedded computer vision techniques.

OpenVX-Based Python Framework for Real-time Cross-Platform Acceleration of Embedded Computer Vision Applications

Embedded real-time vision applications are being rapidly deployed in a large realm of consumer electronics, ranging from automotive safety to surveillance systems. However, the relatively limited computational power of embedded platforms is considered as a bottleneck for many vision applications, necessitating optimization. OpenVX is a standardized interface, released in late 2014, in an attempt to provide both system and kernel level optimization to vision applications. With OpenVX, Vision processing are modeled with coarse-grained data flow graphs, which can be optimized and accelerated by the platform implementer. Current full implementations of OpenVX are given in the programming language C, which does not support advanced programming paradigms such as object-oriented, imperative and functional programming, nor does it have runtime or type-checking. Here we present a python-based full Implementation of OpenVX, which eliminates much of the discrepancies between the object-oriented paradigm used by many modern applications and the native C implementations. Our open-source implementation can be used for rapid development of OpenVX applications in embedded platforms. Demonstration includes static and real-time image acquisition and processing using a Raspberry Pi and a GoPro camera. Code is given as supplementary information. Code project and linked deployable virtual machine are located on GitHub: https://github.com/NBEL-lab/PythonOpenVX.

PRECISION: A Reconfigurable SIMD/MIMD Coprocessor for Computer Vision Systems-on-Chip

Computer vision applications have a large disparity in operations, data representation and memory access patterns from the early vision stages to the final classification and recognition stages. A hardware system for computer vision has to provide high flexibility without compromising performance, exploiting massively spatial-parallel operations but also keeping a high throughput on data-dependent and complex program flows. Furthermore, the architecture must be modular, scalable and easy to adapt to the needs of different applications. Keeping this in mind, a hybrid SIMD/MIMD architecture for embedded computer vision is proposed. It consists of a coprocessor designed to provide fast and flexible computation of demanding image processing tasks of vision applications. A 32-bit 128-unit device was prototyped on a Virtex-6 FPGA which delivers a peak performance of 19.6 GOP/s and 7.2 W of power dissipation.

Dataflow object detection system for FPGA‐based smart camera

Embedded computer vision based smart systems raise challenging issues in many research fields, including real-time vision processing, communication protocols or distributed algorithms. The amount of data generated by cameras using high resolution image sensors requires powerful computing systems to be processed at digital video frame rates. Consequently, the design of efficient and flexible smart cameras, with on-board processing capabilities, has become a key issue for the expansion of smart vision systems relying on decentralised processing at the image sensor node level. In this context, field programmable gate arrays (FPGA)-based platforms, supporting massive data parallelism, offer large opportunities to match real-time processing constraints compared with platforms based on general purpose processors. In this study, the authors describe the implementation, on such a platform, of a configurable object detection application, reformulated according to the dataflow model of computation. The application relies on the computation of the histogram of oriented gradients and a linear SVM-based classification. It is described using the CAPH programming language, allowing efficient hardware descriptions to be generated automatically from high level dataflow specifications without prior knowledge of hardware description languages such as VHDL or Verilog. Results show that the performance of the generated code does not suffer from a significant overhead compared with handwritten HDL code.

Parallelizing Multimodal Background Modeling on a Low-Power Integrated GPU

Background modeling techniques for embedded computer vision applications must balance accuracy, speed, and power. Basic background modeling techniques run quickly, but their accuracy is not sufficient for computer vision problems involving dynamic background. In contrast, adaptive background modeling techniques are more robust, but run more slowly. Due to its high inherent fine-grain parallelism, robust adaptive background modeling has been implemented on GPUs with significant performance improvements over CPUs. However, these implementations are infeasible in embedded applications due to the high power ratings of the targeted general-purpose GPU platforms. This paper focuses on exploiting fine-grain data parallelism and optimizing memory access patterns to target a low-cost adaptive background modeling algorithm multimodal mean (MMM) to a low-power GPU with thermal design power (TDP) of only 12 watts. The algorithm has comparable accuracy with the Gaussian mixture model (GMM) algorithm, but less computational and memory cost. It achieves a frame rate of 392 fps with a full VGA resolution (640x480) frame on the low-power integrated GPU NVIDIA ION. This is a 20x speed-up of the MMM algorithm compared to the embedded CPU platform Intel Atom of comparable TDP. In addition, the MMM algorithm attains a 5-6x speed up over the GMM implementation on the ION GPU platform.

PULP: A Ultra-Low Power Parallel Accelerator for Energy-Efficient and Flexible Embedded Vision

Novel pervasive devices such as smart surveillance cameras and autonomous micro-UAVs could greatly benefit from the availability of a computing device supporting embedded computer vision at a very low power budget. To this end, we propose PULP (Parallel processing Ultra-Low Power platform), an architecture built on clusters of tightly-coupled OpenRISC ISA cores, with advanced techniques for fast performance and energy scalability that exploit the capabilities of the STMicroelectronics UTBB FD-SOI 28nm technology. We show that PULP performance can be scaled over a 1x-354x range, with a peak theoretical energy efficiency of 211 GOPS/W. We present performance results for several demanding kernels from the image processing and vision domain, with post-layout power modeling: a motion detection application that can run at an efficiency up to 192 GOPS/W (90 % of the theoretical peak); a ConvNet-based detector for smart surveillance that can be switched between 0.7 and 27fps operating modes, scaling energy consumption per frame between 1.2 and 12mJ on a 320 ×240 image; and FAST + Lucas-Kanade optical flow on a 128 ×128 image at the ultra-low energy budget of 14 μJ per frame at 60fps.

Tailoring design for embedded computer vision applications

Numerical dynamic range analysis combines with architectural simulation to help designers explore algorithmic and architectural tradeoffs that can meet embedded computer vision's demand for precision and high-memory bandwidth.