Vision-based Sensor Research Articles

An evaluation of a vision-based sensor performance in Huanglongbing disease identification

Huanglongbing (HLB) disease is a critical infection which has dangerously affected the citrus production in Florida and has also been observed in California and Texas. No active treatment for this infection has been reported yet and the HLB infected tree will eventually die. However, early identification and removal of the HLB affected trees will secure the healthy trees in the grove. Accumulation of starch on infected leaves is an early symptom of HLB. Starch can rotate the polarisation plane of light in a certain waveband. A customised vision sensor was developed based on this characteristic to identify the HLB symptom. The vision sensor images were compared with the images captured by a colour camera to demonstrate the improvement achieved in this method. Also, the starch accumulation identification was studied for citrus leaves before and after being ground. The results showed an enhanced HLB identification performance using the developed vision sensor. • Accumulated starch can rotate the polarisation plane of light near 600 nm. • The customised vision sensor optimised the HLB infection identification. • Starch accumulation was more evident in ground citrus leaf. • The vision sensor could separate HLB infection from nutrient deficiency.

Accurate fibre orientation measurement for carbon fibre surfaces

Carbon- and glass fibre materials exhibit challenging optical properties, in particular highly specular reflectivity. State of the art vision-based sensor systems use diffuse light setups to suppress specular reflections and apply texture analysis to obtain fibre orientations for surface patches.We investigate the reflection behaviour of fibrous materials. The derived reflection model is the basis for a sensor system that directly measures fibre orientation as well as diffuse and specular reflectivity per pixel. The proposed sensor is robust to changes of the fibre material and capable to deliver fast and accurate information about fibre direction without time consuming texture analysis. An empirical evaluation shows that the root mean squared error of calculated fibre angles is around or below 1°.

Appearance-based multimodal human tracking and identification for healthcare in the digital home.

There is an urgent need for intelligent home surveillance systems to provide home security, monitor health conditions, and detect emergencies of family members. One of the fundamental problems to realize the power of these intelligent services is how to detect, track, and identify people at home. Compared to RFID tags that need to be worn all the time, vision-based sensors provide a natural and nonintrusive solution. Observing that body appearance and body build, as well as face, provide valuable cues for human identification, we model and record multi-view faces, full-body colors and shapes of family members in an appearance database by using two Kinects located at a home's entrance. Then the Kinects and another set of color cameras installed in other parts of the house are used to detect, track, and identify people by matching the captured color images with the registered templates in the appearance database. People are detected and tracked by multisensor fusion (Kinects and color cameras) using a Kalman filter that can handle duplicate or partial measurements. People are identified by multimodal fusion (face, body appearance, and silhouette) using a track-based majority voting. Moreover, the appearance-based human detection, tracking, and identification modules can cooperate seamlessly and benefit from each other. Experimental results show the effectiveness of the human tracking across multiple sensors and human identification considering the information of multi-view faces, full-body clothes, and silhouettes. The proposed home surveillance system can be applied to domestic applications in digital home security and intelligent healthcare.

Distributed collaborative 3D pose estimation of robots from heterogeneous relative measurements: an optimization on manifold approach

SUMMARYWe propose a distributed algorithm for estimating the 3D pose (position and orientation) of multiple robots with respect to a common frame of reference when Global Positioning System is not available. This algorithm does not rely on the use of any maps, or the ability to recognize landmarks in the environment. Instead, we assume that noisy relative measurements between pairs of robots are intermittently available, which can be any one, or combination, of the following: relative pose, relative orientation, relative position, relative bearing, and relative distance. The additional information about each robot's pose provided by these measurements are used to improve over self-localization estimates. The proposed method is similar to a pose-graph optimization algorithm in spirit: pose estimates are obtained by solving an optimization problem in the underlying Riemannian manifold $(SO(3)\times{\mathcal R}^3)^{n(k)}$. The proposed algorithm is directly applicable to 3D pose estimation, can fuse heterogeneous measurement types, and can handle arbitrary time variation in the neighbor relationships among robots. Simulations show that the errors in the pose estimates obtained using this algorithm are significantly lower than what is achieved when robots estimate their pose without cooperation. Results from experiments with a pair of ground robots with vision-based sensors reinforce these findings. Further, simulations comparing the proposed algorithm with two state-of-the-art existing collaborative localization algorithms identify under what circumstances the proposed algorithm performs better than the existing methods. In addition, the question of trade-offs between cost (of obtaining a certain type of relative measurement) and benefit (improvement in localization accuracy) for various types of relative measurements is considered.

Optical stimulator for vision-based sensors

Abstract We have developed an optical stimulator system for vision-based sensors. The stimulator is an efficient tool for stimulating a camera during on-ground testing with scenes representative of spacecraft flights. Such scenes include starry sky, planetary objects, and other spacecraft. The optical stimulator is used as a test bench to simulate high-precision navigation by different types of camera systems that are used onboard spacecraft, planetary rovers, and for spacecraft rendezvous and proximity maneuvers. Careful hardware design and preoperational calibration of the stimulator result in high precision and long-term stability. The system can be continuously used over several days. By facilitating a full camera including optics in the loop, the stimulator enables the more realistic simulation of flight maneuvers based on navigation cameras than pure computer simulations or camera stimulations without the involvement of the actual optics.

Autonomous vision-based detection of non-stellar objects flying in formation with camera point of view

The μ-advanced stellar compass (μASC) is a camera system which autonomously can provide attitude information based on the visible stars. An extended functionality of the μASC, called the vision-based sensor (VBS), enables the camera system to render rendezvous and docking navigation information for a formation flying spacecraft. The VBS system facilitates different operation modes, enabling i.e. inertial line-of-sight (LoS) solutions for any spacecraft reflecting more light than an Mv7 star. This paper will focus on the LoS solutions of the VBS system, with a thorough analysis of inflight performance and future mission aspects. The captured data used for the analysis presented in this paper, includes centroids from the VBS camera head unit (CHU) dedicated for inertial LoS operation and from the two standard startracker CHUs. By combining the data from these three CHUs, a total of 300 hours of operation distributed over the entire year is included in the thorough data analysis and performance characterisation.

Acquisition of Contact Force and Slippage Using a Vision-Based Tactile Sensor with a Fluid-Type Touchpad for the Dexterous Handling of Robots

This paper presents a new approach for estimating contact force and slippage by using a vision-based tactile sensor with a fluid-type touchpad for the dexterous handling of robots. The sensor consists of a CCD camera, LED lights, a transparent acrylic plate and a deformable touchpad. The sensor can obtain a variety of tactile information, such as the contact force, shape, contact region, position and orientation of an object in contact with the fluid-type touchpad. The previous method for measuring contact force requires extensive calculation; this paper proposes a new method based on lookup tables for measuring normal force, tangential force and moment, using a fluid-type touchpad. Additionally, we clarify the mechanism of slippage between a fluid-type touchpad and a contacted object. This mechanism is efficient for applying the proposed slippage estimation method to a fluid-type touchpad. The validation of the proposed methods is confirmed in the experimental results.

Vision-Based Sensing in Dynamic Tests

The availability of a suitable data acquisition sensor network is a key implementation issue to link models with real world structures. Non-contact displacement sensors should be preferred since they do not change the system properties. A two-dimensional vision-based displacement measurement sensor is the focus of this contribution. In particular, the perspective distortion introduced by the angle between the optic axis of the camera and the normal to the plane in which the structural system deforms is considered. A two-dimensional affine transformation is utilized to eliminate the distortion from the recorded to the distortion-free image. The results of a laboratory experiment show the potential of the proposed approach.

Low-cost Sensors Data Fusion for Small Size Unmanned Aerial Vehicles Navigation and Guidance

A new integrated navigation system designed for small size Unmanned Aerial Vehicles (UAVs) is presented. The proposed system is based on a number of low-cost avionics sensors, including Global Navigation Satellite Systems (GNSS), Micro-Electro-Mechanical System (MEMS) based Inertial Measurement Unit (IMU) and Vision Based Sensors (VBS). The use of an Aircraft Dynamics Models (ADMs) to provide additional information to compensate for the shortcomings of Vision Based Navigation (VBN) and MEMS-IMU sensors in high-dynamics attitude determination tasks is also considered. Additionally, the research concentrates on the potential of carrier-phase GNSS for Attitude Determination (GAD) using interferometric techniques. The main objective is to design a compact, light and relatively inexpensive system capable of providing the required navigation performance (position and attitude data) in all phases of flight of small UAVs, with a special focus on precision approach and landing, where VBN techniques can be fully exploited in a multi-sensor data fusion architecture. An Extended Kalman Filter (EKF) is developed to integrate the information provided by the different sensors and to provide estimates of position, velocity and attitude of the UAV platform in real-time. Three different integrated navigation system architectures are implemented. The first architecture uses VBN at 20 Hz and GNSS at 1 Hz to augment the MEMS-IMU running at 100 Hz. The second mode also includes the ADM (computations performed at 100 Hz) to provide augmentation of the attitude channel. The third fusion architecture uses GNSS based attitude values. The simulations are carried out on the AEROSONDE UAV performing high-dynamics manoeuvres repre-sentative of the UAV operational flight envelope. Simulation of the VBN-IMU-GNSS (VIG) integrated navigation system shows that the system can attain position, velocity and attitude accuracies complying with Category Two (CAT II) precision approach requirements. Simulation of the VBN-IMU-GNSS-ADM (VIGA) system also shows promising results, since the achieved attitude accuracy is higher using the ADM-VBN-IMU than using VBN-IMU only. However, due to rapid divergence of the ADM virtual sensor, there is a need for frequent re-initialisation of the ADM data module, which is strongly dependent on the UAV flight dynamics and the specific manoeuvring transitions performed. In the simulation of the third integrated navigation system, the VIG system is augmented by employing the GAD, forming the VIG-GAD (VIGGA) system architecture. The performances achieved with the VIG, VIGA and VIGGA integrated Navigation and Guidance System (NGS) are presented and are in line with the International Civil Aviation Organization (ICAO) precision approach requirements.

Validity, Reliability, and Sensitivity of a 3D Vision Sensor-based Upper Extremity Reachable Workspace Evaluation in Neuromuscular Diseases

Introduction: One of the major challenges in the neuromuscular field has been lack of upper extremity outcome measures that can be useful for clinical therapeutic efficacy studies. Using vision-based sensor system and customized software, 3-dimensional (3D) upper extremity motion analysis can reconstruct a reachable workspace as a valid, reliable and sensitive outcome measure in various neuromuscular conditions where proximal upper extremity range of motion and function is impaired. Methods: Using a stereo-camera sensor system, 3D reachable workspace envelope surface area normalized to an individual’s arm length (relative surface area: RSA) to allow comparison between subjects was determined for 20 healthy controls and 9 individuals with varying degrees of upper extremity dysfunction due to neuromuscular conditions. All study subjects were classified based on Brooke upper extremity function scale. Right and left upper extremity reachable workspaces were determined based on three repeated measures. The RSAs for each frontal hemi-sphere quadrant and total reachable workspaces were determined with and without loading condition (500 gram wrist weight). Data were analyzed for assessment of the developed system and validity, reliability, and sensitivity to change of the reachable workspace outcome. Results: The mean total RSAs of the reachable workspace for the healthy controls and individuals with NMD were significantly different (0.586 ± 0.085 and 0.299 ± 0.198 respectively; p<0.001). All quadrant RSAs were reduced for individuals with NMDs compared to the healthy controls and these reductions correlated with reduced upper limb function as measured by Brooke grade. The upper quadrants of reachable workspace (above the shoulder level) demonstrated greatest reductions in RSA among subjects with progressive severity in upper extremity impairment. Evaluation of the developed outcomes system with the Bland-Altman method demonstrated narrow 95% limits of agreement (LOA) around zero indicating high reliability. In addition, the intraclass correlation coefficient (ICC) was 0.97. Comparison of the reachable workspace with and without loading condition (wrist weight) showed significantly greater RSA reduction in the NMD group than the control group (p<0.012), with most of the workspace reduction occurring in the ipsilateral upper quadrant relative to the tested arm (p<0.001). Reduction in reachable workspace due to wrist weight was most notable in those subjects with NMD with marginal strength reserve and moderate degree of impairment (Brooke = 2) rather than individuals with mild upper extremity impairment (Brooke = 1) or individuals who were more severely impaired (Brooke =3). Discussion: The developed reachable workspace evaluation method using scalable 3D vision technology appears promising as an outcome measure system for clinical studies. A rationally-designed combination of upper extremity outcome measures including a region-specific global upper extremity outcome measure, such as the reachable workspace, complemented by targeted disease- or function-specific endpoints, may be optimal for future clinical efficacy trials.

Viability control for a class of underactuated systems

This paper addresses the feedback control design for a class of nonholonomic systems which are subject to inequality state constraints defining a constrained (viability) set K. Based on concepts from viability theory, the necessary conditions for selecting viable controls for a nonholonomic system are given, so that system trajectories starting in K always remain in K. Furthermore, a class of state feedback control solutions for nonholonomic systems are redesigned by means of switching control, so that system trajectories starting in K converge to a goal set G in K, without ever leaving K. The proposed approach can be applied in various problems, whose objective can be recast as controlling a nonholonomic system so that the resulting trajectories remain forever in a subset K of the state space, until they converge into a goal (target) set G in K. The motion control for an underactuated marine vehicle in a constrained configuration set K is treated as a case study; the set K essentially describes the limited sensing area of a vision-based sensor system, and viable control laws which establish convergence to a goal set G in K are constructed. The robustness of the proposed control approach under a class of bounded external perturbations is also considered. The efficacy of the methodology is demonstrated through simulation results.

Gait disorder rehabilitation using vision and non-vision based sensors: A systematic review

Even though the amount of rehabilitation guidelines has never been greater, uncertainty continues to arise regarding the efficiency and effectiveness of the rehabilitation of gait disorders. This question has been hindered by the lack of information on accurate measurements of gait disorders. Thus, this article reviews the rehabilitation systems for gait disorder using vision and non-vision sensor technologies, as well as the combination of these. All papers published in the English language between 1990 and June, 2012 that had the phrases "gait disorder", "rehabilitation", "vision sensor", or "non vision sensor" in the title, abstract, or keywords were identified from the SpringerLink, ELSEVIER, PubMed, and IEEE databases. Some synonyms of these phrases and the logical words "and", "or", and "not" were also used in the article searching procedure. Out of the 91 published articles found, this review identified 84 articles that described the rehabilitation of gait disorders using different types of sensor technologies. This literature set presented strong evidence for the development of rehabilitation systems using a markerless vision-based sensor technology. We therefore believe that the information contained in this review paper will assist the progress of the development of rehabilitation systems for human gait disorders.

Vision-Based Sensor for Early Detection of Periodical Defects in Web Materials

During the production of web materials such as plastic, textiles or metal, where there are rolls involved in the production process, periodically generated defects may occur. If one of these rolls has some kind of flaw, it can generate a defect on the material surface each time it completes a full turn. This can cause the generation of a large number of surface defects, greatly degrading the product quality. For this reason, it is necessary to have a system that can detect these situations as soon as possible. This paper presents a vision-based sensor for the early detection of this kind of defects. It can be adapted to be used in the inspection of any web material, even when the input data are very noisy. To assess its performance, the sensor system was used to detect periodical defects in hot steel strips. A total of 36 strips produced in ArcelorMittal Avilés factory were used for this purpose, 18 to determine the optimal configuration of the proposed sensor using a full-factorial experimental design and the other 18 to verify the validity of the results. Next, they were compared with those provided by a commercial system used worldwide, showing a clear improvement.

Demonstration of closed-loop airborne sense-and-avoid using machine vision

The ARCAA DSA program has recently demonstrated full closed-loop aircraft sense-and-avoid using vision-based sensor technology. Real-time onboard detection and tracking was accomplished by harnessing the parallel processing capabilities of GPU-based hardware. ARCAA's DSA program aims to develop a cost-effective collision avoidance capability to facilitate the routine operation of UASs in the national airspace as well as enhance manned aviation safety by complementing human pilot “see-and-avoid.

Preface to the special issue on multi-sensor systems in engineering geodesy and navigation

Under the umbrella of the IAG (International Association of Geodesy) the symposium ‘Multi-sensor Systems for Engineering Geodesy’ was held at the XXV General Assembly of the International Union of Geodesy and Geophysics (IUGG) at the end of June, early July 2011 in Melbourne, Australia. The convenors of this symposium were Dorota Grejner-Brzezinska and Hans-Joerg Kutterer. The symposium included two oral sessions with in total ten presentations and one poster session with seven posters. In this issue of the journal four revised papers from the conference are presented, namely the first, fourth, fifth and sixth paper. In addition, two papers were selected which deal with the topic of the use, and their integration, of GNSS and multi-sensors in engineering geodesy and navigation applications. Many applications in engineering geodesy and navigation nowadays rely on GNSS positioning. Due to its limitations the use of multi-sensors as an alternative and/or backup to GNSS positioning has become an increasingly important field of research. In the coming years the IAG Sub-Commission 4.1 ‘Alternatives and Backups to GNSS’ under Commission 4 ‘Positioning and Applications’ will place special emphasis on research and other activities that address multi-sensor system theory and applications. Important topics include integrated guidance, navigation, positioning and orientation of airborne and land-based platforms. The primary sensors of interest will be inertial navigation systems (INS); however, the important role of other emerging techniques, such as pseudolites, vision-based sensors, ultrasonic and magnetic sensors etc., used in navigation, engineering geodesy and environmental monitoring is also recognised. New concepts and innovations in navigation algorithms, sensor calibration, synchronization and inter-calibration, real-time sensor information processing, sensor and data fusion, and automation techniques for information extraction from multi-sensor systems using expert systems continue to be investigated. Furthermore the progress in new applications (not limited to conventional engineering geodesy and navigation) of multi-sensor systems (e.g., in transportation, engineering, car navigation, environmental monitoring personal navigation, indoor navigation) is an important research focus in the years to come.

Control of an Underactuated Underwater Vehicle in 3D Space under Field-of-View Constraints

This paper presents a state feedback control approach for the motion of an under-actuated, pitch-capable Unmanned Underwater Vehicle (UUV) in a 3-dimensional workspace. The objective is to steer the vehicle to a desired configuration with respect to (w.r.t.) a target of interest, in the presence of field-of-view (f.o.v.) constraints due to a vision-based sensor system. The control design is based on dipolar reference vector fields and tools from viability theory, and guarantees the convergence of the vehicle in a neighborhood of the desired configuration, without violating the f.o.v. constraints. The unactuated degrees of freedom (d.o.f.) are shown to be input-to-state stable (ISS). Simulations demonstrate the efficacy of the approach.

Vision-based force sensing for nanomanipulation

In this paper, a vision-based algorithm for estimating tip interaction forces on a deflected atomic orce microscope (AFM) cantilever is described. Specifically, we propose that the algorithm can estimate forces acting on an AFM cantilever being used as a nanomanipulator inside a scanning electron microscope (SEM). The vision-based force sensor can provide force feedback in real time, a feature absent in many SEMs. A methodology based on cantilever slope detection is used to estimate the forces acting on the cantilever tip. The technique was tested on a scaled model of the nanoscale AFM cantilever and verified using theoretical estimates as well as direct strain measurements. Artificial SEM noise was introduced in the macroscale images to characterize our sensor under varying levels of noise and other SEM effects. Prior knowledge about the cantilever is not required, and the algorithm runs independent of human input. The method is shown to be effective under varying noise levels, and demonstrates improving performance as magnification levels are decreased. Therefore, we conclude that the vision-based force sensing algorithm is best suited for continuous operation of the SEM, fast scanning rates, and large fields-of-view associated with low magnification levels.

An image differencing method for interface level detection in separation cells

Bitumen extraction from oil sands is carried out in large separation cells using a water-based extraction process. Controlling the Bitumen-froth and Middlings interface in these cells at an optimum level provides significant economical and environmental benefits. Traditional sensors are not reliable in estimating this interface level and novel vision-based sensors have been developed previously to overcome this problem. These sensors estimate the interface level and its confidence for separation cells with a single sight glass. The confidence value only represents the turbidity of the interface and hence cannot be used for control decision making in all process conditions. The current work describes an image differencing algorithm for interface level detection which also facilitates the computation of a confidence estimate that is accurate in most process situations. The confidence value is computed based on noise statistics, an appropriately chosen edge detection method and a change detection algorithm. Another significant advantage of the algorithm is that both the interface level and the confidence estimation procedures can be extended in a straight-forward manner to handle the presence of multiple sight glasses. Off-line results show that the algorithm accurately detects the interface level in normal process conditions (with high confidence values) and outputs correct confidence values in other situations with very low false-positive and false-negative error rates. On-line industrial implementation results show that the vision sensor tracks the interface level very closely and results in significant automation of plant.

Synchronisation of vision-based sensor networks with variable frame rates

This paper introduces a synchronisation methodology for distributed vision-based sensor networks with clock skew or variable frame rates. The methodology requires a ballistic or otherwise predictive object to be tracked by the sensor network and used to calibrate the clock skew and/or relative variable frame rates between the cameras. The relative time stamp of each image captured can be extracted using the dynamic model of the predictive object. The time stamps and a best fit correlation are used to synchronise all the cameras (and their video streams) that are tracking the same predictive object. In sport the predictive object is likely to be a ballistic object such as a ball or a player in flight, while in security applications, the predictive object may be trains, cars or other objects travelling at predictable speeds in known locations.

Finger-Shaped GelForce: Sensor for Measuring Surface Traction Fields for Robotic Hand.

It is believed that the use of haptic sensors to measure the magnitude, direction, and distribution of a force will enable a robotic hand to perform dexterous operations. Therefore, we develop a new type of finger-shaped haptic sensor using GelForce technology. GelForce is a vision-based sensor that can be used to measure the distribution of force vectors, or surface traction fields. The simple structure of the GelForce enables us to develop a compact finger-shaped GelForce for the robotic hand. GelForce that is developed on the basis of an elastic theory can be used to calculate surface traction fields using a conversion equation. However, this conversion equation cannot be analytically solved when the elastic body of the sensor has a complicated shape such as the shape of a finger. Therefore, we propose an observational method and construct a prototype of the finger-shaped GelForce. By using this prototype, we evaluate the basic performance of the finger-shaped GelForce. Then, we conduct a field test by performing grasping operations using a robotic hand. The results of this test show that using the observational method, the finger-shaped GelForce can be successfully used in a robotic hand.