Real-time Computer Vision System Research Articles

A scalable real-time computer vision system for student posture detection in smart classrooms

A scalable real-time computer vision system for student posture detection in smart classrooms

Hand Cricket Game Application Using Computer Vision

Hand cricket is an attempt to experience the childhood entrainment of playing cricket using our hands. This paper presents an application built to play Hand cricket using a computer vision-based real-time 3D hand gesture recognition system. To play the game the user will give inputs using hand gestures. These hand gestures will be identified using a real-time computer vision system that runs based on a CNN model. Since in this game both players must play simultaneously the computer also comes up with a number from 1 to 6 when a hand gesture is being read and recognized. The core engine of building this app is an image classifier that classifies a picture of a hand into 1,2,3,4,5,6 and none. This paper shows an interactive simple game application developed, that tries to make the game feel more natural and as if playing with a real person.

A Manycore Vision Processor for Real-Time Smart Cameras.

Real-time image processing and computer vision systems are now in the mainstream of technologies enabling applications for cyber-physical systems, Internet of Things, augmented reality, and Industry 4.0. These applications bring the need for Smart Cameras for local real-time processing of images and videos. However, the massive amount of data to be processed within short deadlines cannot be handled by most commercial cameras. In this work, we show the design and implementation of a manycore vision processor architecture to be used in Smart Cameras. With massive parallelism exploration and application-specific characteristics, our architecture is composed of distributed processing elements and memories connected through a Network-on-Chip. The architecture was implemented as an FPGA overlay, focusing on optimized hardware utilization. The parameterized architecture was characterized by its hardware occupation, maximum operating frequency, and processing frame rate. Different configurations ranging from one to eighty-one processing elements were implemented and compared to several works from the literature. Using a System-on-Chip composed of an FPGA integrated into a general-purpose processor, we showcase the flexibility and efficiency of the hardware/software architecture. The results show that the proposed architecture successfully allies programmability and performance, being a suitable alternative for future Smart Cameras.

Stabilized adaptive sampling control for reliable real-time learning-based surveillance systems

In modern security systems such as CCTV-based surveillance applications, real-time deep-learning based computer vision algorithms are actively utilized for always-on automated execution. The real-time computer vision system for surveillance applications is highly computation-intensive and exhausts computation resources when it performed on the device with a limited amount of resources. Based on the nature of Internet-of-Things networks, the device is connected to main computing platforms with offloading techniques. In addition, the real-time computer vision system such as the CCTV system with image recognition functionality performs better when arrival images are sampled at a higher rate because it minimizes missing video frame feeds. However, performing it at overwhelmingly high rates exposes the system to the risk of a queue overflow that hampers the reliability of the system. In order to deal with this issue, this paper proposes a novel queue-aware dynamic sampling rate adaptation algorithm that optimizes the sampling rates to maximize the computer vision performance(i.e., recognition ratio) while avoiding queue overflow under the concept of Lyapunov optimization framework. Through extensive system simulations, the proposed approaches are shown to provide remarkable gains.

Real-time computer vision system for tracking simultaneously subject-specific rigid head and non-rigid facial mimic movements using a contactless sensor and system of systems approach

Background and ObjectiveHead and facial mimic animations play important roles in various fields such as human-machine interactions, internet communications, multimedia applications, and facial mimic analysis. Numerous studies have been trying to simulate these animations. However, they hardly achieved all requirements of full rigid head and non-rigid facial mimic animations in a subject-specific manner with real-time framerates. Consequently, this present study aimed to develop a real-time computer vision system for tracking simultaneously rigid head and non-rigid facial mimic movements. MethodsOur system was developed using the system of systems approach. A data acquisition sub-system was implemented using a contactless Kinect sensor. A subject-specific model generation sub-system was designed to create the geometrical model from the Kinect sensor without texture information. A subject-specific texture generation sub-system was designed for enhancing the reality of the generated model with texture information. A head animation sub-system with graphical user interfaces was also developed. Model accuracy and system performances were analyzed. ResultsThe comparison with MRI-based model shows a very good accuracy level (distance deviation of ~1 mm in neutral position and an error range of [2–3 mm] for different facial mimic positions) for the generated model from our system. Moreover, the system speed can be optimized to reach a high framerate (up to 60 fps) during different head and facial mimic animations. ConclusionsThis study presents a novel computer vision system for tracking simultaneously subject-specific rigid head and non-rigid facial mimic movements in real time. In perspectives, serious game technology will be integrated into this system towards a full computer-aided decision support system for facial rehabilitation.

AIR-Chem: Authentic Intelligent Robotics for Chemistry.

The new era with prosperous artificial intelligence (AI) and robotics technology is reshaping the materials discovery process in a more radical fashion. Here we present authentic intelligent robotics for chemistry (AIR-Chem), integrated with technological innovations in the AI and robotics fields, functionalized with modules including gradient descent-based optimization frameworks, multiple external field modulations, a real-time computer vision (CV) system, and automated guided vehicle (AGV) parts. AIR-Chem is portable and remotely controllable by cloud computing. AIR-Chem can learn the parametric procedures for given targets and carry on laboratory operations in standalone mode, with high reproducibility, precision, and availability for knowledge regeneration. Moreover, an improved nucleation theory of size focusing on inorganic perovskite quantum dots (IPQDs) is theoretically proposed and experimentally testified to by AIR-Chem. This work aims to boost the process of an unmanned chemistry laboratory from the synthesis of chemical materials to the analysis of physical chemical properties, and it provides a vivid demonstration for future chemistry reshaped by AI and robotics technology.

Software and Hardware Implementation of Video Camera, Compatible with GigE Vision Standard

Different variants of video streaming technologies are considered. From the standpoint of locally distributed computer vision systems implementation, the GigE Vision technology is selected for further investigation, since it is capable of providing relatively long channel distances, high robustness and image quality, low latency of video data transmission, and uses standard connection interfaces available in most of computers. Thereby the implementation of own GigE Vision compatible camera is chosen as the particular objective of the paper. Two ways of achieving of this objective are considered. The fists one is based on software implementation solely, and uses Aravis open source library, which is coupled with the Video4Linux API. It is shown that this software solution is capable of video streaming, but has high transmission delays, making it inexpedient for use in real-time computer vision systems. On the basis of experiments and the analysis of channel load, it was found that in the test system, the delays were caused mainly by the Video4Linux library. To overcome this problem, the second variant of camera, which uses the hardware implementation of primary video capturing part, was proposed. It is based on system-on-chip (SoC) solution, which incorporates FPGA and ARM-processor. In this system, the video sensor is connected to the FPGA. In the FPGA, the control of video sensor and the special direct memory access controller are implemented. As in the first camera variant, the Aravis library is responsible for the data transmission over Ethernet (it “resides” in the ARM-processor part of SoC). The direct memory access controller, which is implemented in the FPGA part, fills a dedicated buffer in ARM-processor’s RAM, taking into account the peculiarities of data transmission over the network. Particularly, data are sent using UDP protocol by portions, thus the controller includes the respective module, which generates interrupt requests immediately, when some fraction of pixels are loaded from the video sensor to the buffer. This fraction can be optimally estimated from the properties of network connection and is controlled by the Aravis library. Such a structure of direct memory access controller does not require waiting of a full frame transferring from the video sensor (which appears in Video4Linux API) and consequently allow to perform pipelined processing, which leads to minimization of data transmission delays. Ref. 10, fig. 3.

DETECTION AND TRACKING OF MOVING OBJECTS WITH REAL-TIME ONBOARD VISION SYSTEM

Abstract. Detection of moving objects in video sequence received from moving video sensor is a one of the most important problem in computer vision. The main purpose of this work is developing set of algorithms, which can detect and track moving objects in real time computer vision system. This set includes three main parts: the algorithm for estimation and compensation of geometric transformations of images, an algorithm for detection of moving objects, an algorithm to tracking of the detected objects and prediction their position. The results can be claimed to create onboard vision systems of aircraft, including those relating to small and unmanned aircraft.

Обработка зрительных данных в системе технического зрения реального времени с использованием ПЛИС

Обработка зрительных данных в системе технического зрения реального времени с использованием ПЛИС

Effects of acoustic vibration on the reorientations of <italic>C. elegans</italic>

As a typical model organism, the nematode C. elegans detect and respond to diverse environmental stimuli, such as light, temperature, odors and chemicals. The underline neuronal circuitry has been defined by decades of work by C. elegans researchers. However, there’s no work done on its perception to acoustic vibration, which is a critical environmental cue in the natural habitat. For the first time, we quantified worm’s response to acoustic vibration with precisely controlled amplitude and frequency. In an isotropic environment, worm’s navigation can be described as a random walk. Periods of forward movements are interrupted by random reorientations. C. elegans respond to environmental cues by biasing its reorientations. In this work, we delivered acoustic wave with defined amplitude and frequency to freely moving C. elegans . A custom-built real-time computer vision system was used to record the movements of individual young adult worms navigating the agar surface. Data collected was analyzed using customized particle-tracking and shape analysis algorithms. To automatically flag reorientations, we considered both the posture of the animal and the movement of its center of mass. Rapid reorientations ( Ω turns or reversal-turns) were flagged when the heading change of the center of mass trajectory was > 60° over 1 second. It was found that worms reorientate more when sound is on and reorientate less when sound is off. To quantify animal’s response to acoustic wave, we defined a non-dimensional acoustic sensitive index according to the formula. The index approaches 0 or 1 if there is no response or big response, respectively. We tested animal’s response to acoustic wave of varying displacement amplitude and fixed frequency. It was found that the acoustic sensitive index is proportional to the displacement amplitude. However, when we further tested animal’s response to acoustic wave of fixed displacement amplitude and varying frequency, the data shows that the acoustic sensitive index first increases then decreases with the increase of frequency. To characterize the mechanical vibration of the agar surface during the acoustic stimuli, we used a laser vibrometer (Polytec). It was showed that there’s obvious response when the sound cause vibration is as low as ~200 nm. The vibration level is much lower than other mechanical stimuli, such as gentle hair touch or plate tapping, which were used for studying the mechanical sensation of C. elegans in most labs. Our results suggest that worm can sense acoustic vibration. It responds to acoustic vibration by biasing the reorientation rate during navigation. The acoustic sensitive index depends on the frequency and displacement amplitude of the acoustic wave. The vibration level C. elegans can sense is much lower than people normally thought. Our work laid the ground work for studying C. elegans acoustic sensation. It would be interesting to further understand the neural circuits and cellular mechanism of this behavior.

A real-time multispectral computer vision system for morpho-thermal analysis of footprint plantar

Physical Rehabilitation processes in amputees, especially those immersed in post-conflict policies of the Colombian government, requires, among other things, a full comprehensive biomechanical and clinical study of high quality. Generally, this comprehensive study includes kinetic, kinematic, dynamic analysis, electromyography and thermal and morphological podiatric assessment. Research and technological development around the first four tests have been rapidly so hardware and software tools are flooding the current market. However, podiatric assessment has been a little behind in this aspect, since many clinics and rehabilitation centers prefer conventional methods for footprint plantar analysis, which are invasive, slow, tedious, subjective and poor in visualization. In this work, two new methodologies for obtaining thermal and morphological parameters of the footprint plantar from thermopedographies and podoscopies are proposed respectively. To achieve this, common elements in algorithms for footprint plantar analysis based on image are identified and computer vision techniques are proposed in each case to automate and systematize the whole process. As a result, a computer-assisted with a friendly graphical interface allows a non-expert user obtain and quickly visualize morphological and thermal parameters of the footprint plantar. Some diabetic and non-diabetic patients are analyzed and fusion of morphological and thermal information demonstrates to be a powerful tool for podological diagnosis.

Real-Time Computer Vision System for Continuous Face Detection and Tracking

The ever-increasing number of traffic accidents due to a diminished driver’s vigilance level has become a problem of serious concern to society. With the ever growing traffic conditions, this problem will further deteriorate. For this issue, development of system which can actively monitors driver vigilance level and alert the driver for any insecure driving condition is essential. So this paper gives detailed information about driver vigilance level monitoring system. The ultimate goal of the system is to detect and alert the driver from insecure sleepy or low concentration driving condition. The system consists of two main modules including drivers face and eye detection module and drivers face tracking module. Viola Jones face detection with AdaBoost (AdaptiveBoosting) method and Circular Hough Transform technique are integrated in the drivers face and eye detection module. In the drivers face tracking module, CAMSHIFT (Continuously Adaptive Mean Shift) algorithm has been used for continuous face tracking of driver. The main components of the system consist of a video camera, a specially designed hardware system based on Raspberry Pi for real-time image processing and controlling the alarm system. In the proposed system, only one video camera is used in practice yet an achievement of fast and accurate detection results are obtained.

A semi-automated motion-tracking analysis of locomotion speed in the C. elegans transgenics overexpressing beta-amyloid in neurons.

Multi-Worm Tracker (MWT) is a real-time computer vision system that can simultaneously quantify motional patterns of multiple worms. MWT provides several behavioral parameters, including analysis of accurate real-time locomotion speed in the nematode, Caenorhabditis elegans. Here, we determined locomotion speed of the Alzheimer's disease (AD) transgenic strain that over-expresses human beta-amyloid1-42 (Aβ) in the neurons. The MWT analysis showed that the AD strain logged a slower average speed than the wild type (WT) worms. The results may be consistent with the observation that the AD patients with dementia tend to show deficits in physical activities, including frequent falls. The AD strain showed reduced ability of the eggs to hatch and slowed hatching of the eggs. Thus, over-expression of Aβ in neurons causes negative effects on locomotion and hatchability. This study sheds light on new examples of detrimental effects that Aβ deposits can exhibit using C. elegans as a model system. The information gathered from this study indicates that the motion tracking analysis is a cost-effective, efficient way to assess the deficits of Aβ over-expression in the C. elegans system.

Continuous real-time monitoring and neural network modeling of apple slices color changes during hot air drying

A hot air drying system equipped with real-time computer vision system was used to investigate the effects of drying variables on apple slices color changes. Drying experiments were conducted at drying air temperatures of 50–70°C, drying air velocities of 1–2m/s, and samples thicknesses of 2–6mm. A multilayer perceptron (MLP) artificial neural network (ANN) was also used to correlate color parameters and moisture content of apple slices with drying variables and drying time. The effects of drying air temperature and sample thickness on color changes were dominated over the effect of drying air velocity. However, non-linear and somewhat complex trends were obtained for all color parameters as function of moisture content. The MLP ANN satisfactorily approximated the color and moisture variations of apple slices with correlation coefficient higher than 0.92. Therefore, the computer vision system supplemented with ANN can be used as a non-invasive, low cost, and easy method for fast and in-line assessing and controlling of foodstuffs color and moisture changes during drying.

Development of Computer Vision System to Extract Kansei Information from Facial Expressions

Kansei engineering is one field of engineering where the feelings of humans are applied to manufacturing. If Kansei can be extracted, the design, development, and evaluation of various things are possible. This study aims to build an automatic design and evaluation system for agricultural products and landscapes and then utilize the system for the evaluation and improvement of educational effects. In this presentation, we built a low price and near real-time system to extract Kansei information from the facial expressions of users and obtained knowledge about Kansei. Additionally, we created an algorithm to discriminate Kansei. We built a system to obtain facial expression information of a user from a camera image as our experimental equipment. The characteristic values calculated from them were the variation (or the normalized polygon area change ratio) of the four items; the inner ends of the eyebrows, upper parts of the eyes, lower parts of the eyes, and the corners of the mouth. The target Kansei was three items: positive Kansei (liking, fun, and happy) and negative Kansei (unpleasantness and hatred). The results are as follows:First, for the target three Kansei, changes could be captured through the four specified normalized polygon area change rates. As for the positive Kansei, the changes in the corners of the mouth were greater than those of the others in particular. As for the negative Kansei, the changes in the lower parts of the eyes were slightly greater than those of the others and, as for the Kansei of surprise, the changes in the corners of the mouth were slightly greater. The direction of the change showed a tendency where the positive Kansei increased at the corners of the mouth in particular. The negative Kansei showed a strong tendency to decrease at the inner ends of the eyebrows and the lower parts of the eyes in particular. The Kansei of surprise showed a tendency where the normalized polygon area change rates increased as a whole. Next, we selected one subject because the relationship between the facial expressions and Kansei varies among individuals, and then we built an algorithm to discriminate the three Kansei features. As for the Kansei of positive – negative and negative – surprise, it became possible to discriminate at a rate of nearly 100% from the information of the corners of the mouth. It became possible to discriminate the positive-surprise at a rate of about 80% from the information of the upper parts of the eyes. As described above, although it is necessary to prepare the template image of the seven points of the target person and adjust the parameters after measuring the Kansei facial expression in advance, a low price and near real-time computer vision system to extract facial expression Kansei could be built.

A Real-time Computer Vision System for VehicleTracking and Collision Detection

A Real-time Computer Vision System for VehicleTracking and Collision Detection

A Cutting Line Positioning and Defects Detection System for Polyurethane Plate Based on Computer Vision

In this paper, we present a design of a real-time computer vision system for polyurethane plate cutting line positioning and defects detection. The main defect of polyurethane plate is uneven texture which doesn’t meet the product requirements. We translate the original image to gray image and find the points with strongest gray as the cutting line, extract feathers and detect defects. The experimental results show that it is easy and effective to position cutting points and find defects of polyurethane plate, which can meet the requirements of production and has great practical value.

High-throughput behavioral analysis in C. elegans.

We have designed a real-time computer vision system, the Multi-Worm Tracker (MWT), that can simultaneously quantify the behavior of dozens of Caenorhabditis elegans on a traditional petri plate at video rates. Three traditional behavioral paradigms are examined using this system: spontaneous movement on food, where the behavior changes over tens of minutes; chemotaxis, where turning events must be detected accurately to determine strategy; and habituation of response to tap, where the response is stochastic and changes over time. In each case, manual analysis or automated single-worm tracking would be tedious and time-consuming, but the MWT system allows rapid quantification of behavior with minimal human effort. Thus, this system will enablelarge scale forward and reverse genetic screens for complex behaviors.

Line segment detection using weighted mean shift procedures on a 2D slice sampling strategy

A new line segment detection approach is introduced in this paper for its application in real-time computer vision systems. It has been designed to work unsupervised without any prior knowledge of the imaged scene; hence, it does not require tuning of input parameters. Although many works have been presented on this topic, as far as we know, none of them achieves a trade-off between accuracy and speed as our strategy does. The reduction of the computational cost compared to other fast methods is based on a very efficient sampling strategy that sequentially proposes points on the image that likely belong to line segments. Then, a fast line growing algorithm is applied based on the Bresenham algorithm, which is combined with a modified version of the mean shift algorithm to provide accurate line segments while being robust against noise. The performance of this strategy is tested for a wide variety of images, comparing its results with popular state-of-the-art line segment detection methods. The results show that our proposal outperforms these works considering simultaneously accuracy in the results and processing speed.

Development of a real-time computer vision system for tracking loose-housed pigs

Changes in regulations for livestock animals will in the near future call for loose-house pig breeding systems. These new systems will increase the workload for the farmers, as location and identification of animals will require more time than before. This paper presents a real-time computer vision system for tracking of pigs in loose-housed stables. The system will ease the workload for farmers in identification and locating individual animals. The system consists of a camera and a PC. The PC runs a tracking-algorithm, estimating the positions and identities of the pigs. The tracking algorithm operates in 2 steps. The first step builds up support maps, pointing to preliminary pig segments in each video frame. In the second step the support map segments are used to build up a 5D-Gaussian model of the individual pigs (i.e. position and shape). The system has software correction for fisheye distortion coming from the camera lens. The fisheye lens allows the camera to monitor a much larger area in the stable. The algorithms are developed in MatLab, implemented in C and runs in real-time. Experiments in the lab and in the stable demonstrate the robustness of the system. The system can track at least 3 pigs over a longer time span (more than 8 min) without loosing track and identity of the individual pigs in a realistic experiment.