Real-time Vision System Research Articles

Thin wafer-level camera lenses inspired by insect compound eyes

We propose a microoptical approach to ultra-compact optics for real-time vision systems that are inspired by the compound eyes of insects. The demonstrated module achieves approx. VGA resolution with a total track length of 1.4 mm which is about two times shorter than comparable single-aperture optics on images sensors of the same pixel pitch. The partial images that are separately recorded in different optical channels are stitched together to form a final image of the whole field of view by means of image processing. A software correction is applied to each partial image so that the final image is made free of distortion. The microlens arrays are realized by state of the art microoptical fabrication techniques on wafer-level which are suitable for a potential application in high volume e.g. for consumer electronic products.

Visual navigation with a time-of-flight camera

This paper presents a real-time visual navigation system that uses a time-of-flight camera with no a priori knowledge of the scene. A method of combining time-of-flight and optical cameras is considered. Methods that take color and spatial information into account to increase the localization reliability are considered.

Hand Gesture Recognition for Human-Computer Interaction

Problem statement: With the development of ubiquitous computing, current user interaction approaches with keyboard, mouse and pen are not sufficient. Due to the limitation of these devices the useable command set is also limited. Direct use of hands can be used as an input device for providing natural interaction. Approach: In this study, Gaussian Mixture Model (GMM) was used to extract hand from the video sequence. Extreme points were extracted from the segmented hand using star skeletonization and recognition was performed by distance signature. Results: The proposed method was tested on the dataset captured in the closed environment with the assumption that the user should be in the Field Of View (FOV). This study was performed for 5 different datasets in varying lighting conditions. Conclusion: This study specifically proposed a real time vision system for hand gesture based computer interaction to control an event like navigation of slides in Power Point Presentation.

Video sensor network for real-time traffic monitoring and surveillance

Sensor networks and associated infrastructures become ever more important to the traffic monitoring and control because of the increasing traffic demands in terms of congestion and safety. These systems allow authorities not only to monitor the traffic state at the detection sites, but also to obtain real-time related information (e.g. traffic loads). This study presents a real-time vision system for automatic traffic monitoring based on a network of autonomous tracking units (ATUs) that capture and process images from one or more pre-calibrated cameras. The proposed system is flexible, scalable and suitable for a broad field of applications, including traffic monitoring of tunnels at highways and aircraft parking areas at airports. Another objective of this work is to test and evaluate different image processing and data fusion techniques in order to be incorporated to the final system. The output of the image processing unit is a set of information for each moving object in the scene, such as target ID, position, velocity and classification, which are transmitted to a remote traffic control centre, with remarkably low bandwidth requirements. This information is analysed and used to provide real-time output (e.g. alerts, electronic road signs, ramp meters etc.) as well as to extract useful statistical information (traffic loads, lane changes, average velocity etc.).

The feasibility of an infrared system for real-time visualization and mapping of ultrasound fields

In treatment planning for ultrasound therapy, it is desirable to know the 3D structure of the ultrasound field. However, mapping an ultrasound field in 3D is very slow, with even a single planar raster scan taking typically several hours. Additionally, hydrophones that are used for field mapping are expensive and can be damaged in some therapy fields. So there is value in rapid methods which enable visualization and mapping of the ultrasound field in about 1 min. In this note we explore the feasibility of mapping the intensity distribution by measuring the temperature distribution produced in a thin sheet of absorbing material. A 0.2 mm thick acetate sheet forms a window in the wall of a water tank containing the transducer. The window is oriented at 45° to the beam axis, and the distance from the transducer to the window can be varied. The temperature distribution is measured with an infrared camera; thermal images of the inclined plane could be viewed in real time or images could be captured for later analysis and 3D field reconstruction. We conclude that infrared thermography can be used to gain qualitative information about ultrasound fields. Thermal images are easily visualized with good spatial and thermal resolutions (0.044 mm and 0.05 °C in our system). The focus and field structure such as side lobes can be identified in real time from the direct video output. 3D maps and image planes at arbitrary orientations to the beam axis can be obtained and reconstructed within a few minutes. In this note we are primarily interested in the technique for characterization of high intensity focused ultrasound (HIFU) fields, but other applications such as physiotherapy fields are also possible.

The physiological mirror—a system for unconscious control of a virtual environment through physiological activity

This paper introduces a system for real-time physiological measurement, analysis, and metaphorical visualization within a virtual environment (VE). Our goal is to develop a method that allows humans to unconsciously relate to parts of an environment more strongly than to others, purely induced by their own physiological responses to the virtual reality (VR) displays. In particular, we exploit heart rate, respiration, and galvanic skin response in order to control the behavior of virtual characters in the VE. Such unconscious processes may become a useful tool for storytelling or assist guiding participants through a sequence of tasks in order to make the application more interesting, e.g., in rehabilitation. We claim that anchoring of subjective bodily states to a virtual reality (VR) can enhance a person’s sense of realism of the VR and ultimately create a stronger relationship between humans and the VR.

Real-Time Environmental Monitoring and Notification for Public Safety

Editor's NoteResearchers at Kean University describe their experience developing and deploying a real-time environmental monitoring and visualization system using a mesh network of wireless sensors. Challenges they addressed in their design of the Wireless Integrated Network Sensors (WiNS) application include issues associated with the data volume, latency, and data synchronization from individually clocked sensors.-Doree Duncan Seligmann.

Real-time visual content description system based on MPEG-7 descriptors

This paper presents a real-time Visual Content Description System (VCDS) based on MPEG-7 descriptors. In our approach the system's structure is divided into two parts, the first of which is the extraction of the descriptors using the VCDS. The second part uses the descriptors' values in a particular search algorithm. We propose here original solutions for both parts. The proposed system architecture could be used for real-time video indexing and retrieval, content summarization, content delivery, surveillance, personalized services, etc. The descriptor extractor IP core, which is part of the VCDS, implements four MPEG-7 visual descriptors and was designed for ASIC implementation in CMOS 0.35 μm, which is a novel solution for a real-time content description problem. The proposed hardware architecture splits the computational burden into several threads, so that calculations are made simultaneously in order to improve the system's speed. These methods make the hardware implementation of the most computationally demanding modules of the system more time- and power-efficient. Four different variations of the basic hardware architecture are discussed. New search algorithms based on the VCDS responses are also proposed. Experimental results demonstrate the effectiveness of the hardware architectures, and the new approach to similarity-based searching methods.

Improvement in Recognition Techniques for Human Computer Interaction

Hand gestures are an important modality for human computer interaction (HCI) [1]. Compared to many existing interfaces, hand gestures have the advantages of being easy to use, natural, and intuitive. Successful applications of hand gesture recognition include computer games control [2], human-robot interaction [3], and sign language recognition [4], to name a few. Vision-based recognition systems can give computers the capability of understanding and responding to hand gestures. The usability of such systems greatly depends on their ability to function reliably in common real-world environments, without requiring the user to wear special clothes or cumbersome devices such as colored markers or gloves [4]. The aim of this technique is the proposal of a real time vision system for its application within visual interaction environments through hand gesture recognition, using general-purpose hardware and low cost sensors, like a simple personal computer and an USB web cam, so any user could make use of it in his office or home. The basis of our approach is a fast segmentation process to obtain the moving hand from the whole image, which is able to deal with a large number of hand shapes against different backgrounds and lighting conditions, and a recognition process that identifies the hand posture from the temporal sequence of segmented hands. The use of a visual memory (Stored database) allows the system to handle variations within a gesture and speed up the recognition process through the storage of different variables related to each gesture. A hierarchical gesture recognition algorithm is introduced to recognize a large number of gestures. Three stages of the proposed algorithm are based on a new hand tracking technique to recognize the actual beginning of a gesture using a Kalman filtering process, hidden Markov models and graph matching. Processing time is important in working with large databases. Therefore, special cares are taken to deal with the large number of gestures.

An Architeture with Automatic Load Balancing for Real-Time Simulation and Visualization Systems

An Architeture with Automatic Load Balancing for Real-Time Simulation and Visualization Systems

An Architeture with Automatic Load Balancing for Real-Time Simulation and Visualization Systems

An Architeture with Automatic Load Balancing for Real-Time Simulation and Visualization Systems

A real-time fuzzy reasoning based control system for catching a moving goldfish

Designing a real-time visual tracking system to catch a goldfish is a complex task because of the large amount of streaming video data that must be transmitted and processed immediately when tracking the goldfish. Usually, building such visual servoing systems requires the application of high-cost specialized hardware and the development of complicated visual control software. In this paper, a novel low-cost, real-time visual servo control system is presented. The system uses stereo vision consisting of two calibrated cameras to acquire images of the goldfish, and applies the continuously adaptive mean shift (CAMSHIFT) vision-tracking algorithm to provide feedback of a fish’s real-time position at a high frame rate. It then employs a 5-axis robot manipulator controlled by a fuzzy reasoning system to catch the fish. This visual tracking and servoing system is less sensitive to lighting influences and thus performs more efficiently. Experiments with the proposed method yielded very good results, as the system’s real-time 3D vision successfully tracked two fish and guided the manipulator, which has a net attached to its end effector, to catch one of them.

The Duke University Helicopter Observation Platform

Abstract To establish a helicopter observation platform (HOP), Duke University has purchased a Bell 206 “Jet Ranger” helicopter, which has been equipped with a three-dimensional, high-frequency positioning and attitude-recording system, a data acquisition and real-time visualization system, and high-frequency sensors to measure turbulence, temperature, moisture, and CO2 concentration. Thus, it can collect the variables needed to compute some of the turbulent heat and scalar fluxes (using the eddy correlation technique) at low altitudes and low airspeeds that are not feasible with airplanes, yet are quite valuable for studying the exchanges between the Earth's surface and the atmosphere. The main objective of this paper is to introduce the Duke HOP capability, emphasizing the range of airspeeds at which practically undisturbed air can be sampled in front of its nose, where its sensors and inlets are attached. Analytical, numerical, and observational studies are used for this purpose. In addition to the cur...

The KanNon System - phonemic recognition using Burg-MCE

We have developed a real-time speech visualization system called “KanNon”[1,2] which supports speech communication of hearing-impaired people. The KanNon system presents informations of the speech such as loudness, pitch, sound spectrogram and characters by speech recognition system in real-time. In the present KanNon system, a word-unit speech recongniton system using large scale dictionary is adopted. However, the KanNon system is required quick and simple display of speech contents for smooth communication. For this purpose, we applied phonemic speech recognition system. Also, we have already proposed Japanese 5 vowels (/a/, /i/, /u/, /e/, /o/) recognition methods, applying “Time-Delay Neural Network (TDNN)” [3] and statistical pattern recognition [4].However, correct recognition rate is about 85 percent shown in Tables 1, 2 which is not so high. In this paper, therefore, we attempt to obtain better spectral features for phenemic recognition, we apply the novel spectral estimation method called Burg-MCE[5] method combining Burg method and Minimum Cross Entropy method. We apply human auditory property to power spectrum estimated by Burg-MCE method, and carry out phonemic recognition by using statiscal pattern recognition.

Computer vision system for automated cell pressure probe operation

An automated cell pressure probe was designed to allow robust measurements of cellular water relations without the limitations caused by operator skill levels (rapid response time, good hand–eye coordination, and manual dexterity). The system was based upon a real-time (30 Hz) computer vision system that employed a radial basis function, artificial neural network using direct greyscale template matching for meniscus recognition and control. The system did not require any custom-built hardware but used readily available off-the-shelf components and was designed to be highly portable, scalable, and relatively simple to set up. The system was robust in the presence of debris or dust on or in the glass capillary, and for tapered capillary probes, tolerant of variations in contrast due to changes in the velocity or travel direction of the meniscus. The system was capable of controlling the pressure within ±0.0005 MPa and could perform pressure-clamp experiments at ±0.001 MPa or greater pressure steps. The automatic meniscus system recognition could accurately detect the meniscus location with a mean error of under 6 μm and a standard deviation of less than 3.2 μm. Mean errors in automatic volume estimation ranged from 0.07 to 0.6%, depending upon capillary diameter.

Highlights of the Brazilian Solar Spectroscope

The digital, decimetric (950–2500 MHz) Brazilian Solar Spectroscope (BSS, Sawant, H.S., Subramanian, K.R., Faria, C., et al. Brazilian Solar Spectroscope (BSS). Solar Phys. 200, 167–176, 2001) with high time (10–1000 ms) and frequency (1–10 MHz) resolution is in regular operation since April, 1998, at the National Space Research Institute (INPE) at São José dos Campos, Brazil. The BSS has now been upgraded with a new digital data acquisition and data processing system. The new version of the BSS has improved the observational possibilities with the capability to record up to 200 frequency channels available in the selectable frequency range 950–2500 MHz. The GPS receiver permits the acquisition of data with time accuracy in the order of 0.1 ms. The software system of the BSS is composed by two distinct modules: the first, data acquisition system provides a flexible Graphical User Interface (GUI) that allows one to choose the observational parameters. The second module is the real time visualization system that permits real time visualization of the observed dynamic spectrum and additionally allows procedures for visualization and preliminary analysis of the recorded solar spectra. Using the new visualization system, we have realized two new types of dm-radio fine structures: narrow band type III bursts with positive as well as negative group frequency drift and dots emissions arranged in zebra-like and fiber-like chains. Furthermore, we have found flare generated fast wave trains according to their tadpole signature in wavelet power spectra for a decimetric type IV radio event (June 6, 2000 flare).

Augmented Reality During Robot-assisted Laparoscopic Partial Nephrectomy: Toward Real-Time 3D-CT to Stereoscopic Video Registration

To investigate a markerless tracking system for real-time stereo-endoscopic visualization of preoperative computed tomographic imaging as an augmented display during robot-assisted laparoscopic partial nephrectomy. Stereoscopic video segments of a patient undergoing robot-assisted laparoscopic partial nephrectomy for tumor and another for a partial staghorn renal calculus were processed to evaluate the performance of a three-dimensional (3D)-to-3D registration algorithm. After both cases, we registered a segment of the video recording to the corresponding preoperative 3D-computed tomography image. After calibrating the camera and overlay, 3D-to-3D registration was created between the model and the surgical recording using a modified iterative closest point technique. Image-based tracking technology tracked selected fixed points on the kidney surface to augment the image-to-model registration. Our investigation has demonstrated that we can identify and track the kidney surface in real time when applied to intraoperative video recordings and overlay the 3D models of the kidney, tumor (or stone), and collecting system semitransparently. Using a basic computer research platform, we achieved an update rate of 10 Hz and an overlay latency of 4 frames. The accuracy of the 3D registration was 1 mm. Augmented reality overlay of reconstructed 3D-computed tomography images onto real-time stereo video footage is possible using iterative closest point and image-based surface tracking technology that does not use external navigation tracking systems or preplaced surface markers. Additional studies are needed to assess the precision and to achieve fully automated registration and display for intraoperative use.

Real-time hydro-environmental modeling and visualization system for public engagement

Quantitative environmental impact assessment (EIA) provides a tool for decision makers and the public to understand issues relating to sustainable development. The use of advanced hydro-environmental modelling and visualization technology can help to provide an intuitive and interactive digital platform to better communicate EIA issues and thus promote continuous public involvement and stakeholder engagement. Such a platform must be based on a robust hydrodynamic model engine that provides seamless predictions from the near to far field. This lecture summarizes the key issues of impact assessment in urban coastal cities, and presents examples of impact assessment of several major sewage outfalls, including the Hong Kong Harbour Area Treatment Scheme. The predictions of a recently developed Distributed Entrainment Sink Approach (DESA) for dynamic coupling of the near and far fields are compared with alternative “actual source" approaches and field experiments. The conceptual elements of a real time water quality forecast and management system are outlined.

A Robotic Facial Expression Recognition System Using Real-Time Vision System

The capability of recognizing human facial expression plays an important role in advanced human-robot interaction development. Through recognizing facial expressions, a robot can interact with a user in a more natural and friendly manner. In this paper, we proposed a facial expression recognition system based on an embedded image processing platform to classify different facial expressions on-line in real time. A low-cost embedded vision system has been designed and realized for robotic applications using a CMOS image sensor and digital signal processor (DSP). The current design acquires thirty 640x480 image frames per second (30 fps). The proposed emotion recognition algorithm has been successfully implemented on the real-time vision system. Experimental results on a pet robot show that the robot can interact with a person in a responding manner. The developed image processing platform is effective for accelerating the recognition speed to 25 recognitions per second with an average on-line recognition rate of 74.4% for five facial expressions.

The Heterogeneous Systems Integration Design and Implementation for Lane Keeping on a Vehicle

In this paper, an intelligent automated lane-keeping system is proposed and implemented on our vehicle platform, i.e., TAIWAN i TS-1. This system challenges the online integrating heterogeneous systems such as a real-time vision system, a lateral controller, in-vehicle sensors, and a steering wheel actuating motor. The implemented vision system detects the lane markings ahead of the vehicle, regardless of the varieties in road appearance, and determines the desired trajectory based on the relative positions of the vehicle with respect to the center of the road. To achieve more humanlike driving behavior such as smooth turning, particularly at high levels of speed, a fuzzy gain scheduling (FGS) strategy is introduced to compensate for the feedback controller for appropriately adapting to the SW command. Instead of manual tuning by trial and error, the methodology of FGS is designed to ensure that the closed-loop system can satisfy the crossover model principle. The proposed integrated system is examined on the standard testing road at the Automotive Research and Testing Center (ARTC) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> and extra-urban highways.