Multiple Camera Images Research Articles

A situational understanding enhancer based on augmented visual prompts for teleoperation using a multi-monitor system

This study proposes an operator support system that enhances understandability of multiple monitor for teleoperated disaster response robots. The autonomous environmental camera control system we previously developed could provide operators with various visual information suited to the situation. To enable operators to adequately recognize environmental situations, multiple camera images are projected on multiple monitors, but this may increase the cognitive load of operators. The proposed situational understanding enhancer thus supports selective visual attention, situational awareness, and action decision, based on augmented visual prompts (AVPs). To support both work facilitation and danger avoidance, the AVPs include (i) monitor frame, (ii) obscuration, (iii) end-point vertical arrow, and (iv) distance arrow, with variable color and flashing effect. The system adaptively draws visual attention by changing the types and parameters of the AVPs according to the situations. Debris-removal tasks were conducted using a virtual reality simulator to compare the systems with and without AVP support. The results showed that the AVP support reduced the frequent changes in point of gaze, provided a sense of depth, and drew attention to potential collisions.

Arbitrary Viewpoint Visualization for Teleoperated Hydraulic Excavators

In this paper, we propose a visualization system for the teleoperation of excavation works using a hydraulic excavator. An arbitrary viewpoint visualization system is a visualization system that enables teleoperators to observe the environment around a machine by combining multiple camera images. However, when applied to machines with arms (such as hydraulic excavators), a part of the field of view is shielded by the image of the excavator’s arm; hence, an occlusion occurs behind the arm. Furthermore, it is difficult for teleoperators to understand the three-dimensional (3D) condition of the excavating point because the current system approximates the surrounding environment with a predetermined shape. To solve these problems, we propose two methods: (1) a method to reduce the occluded region and expand the field of view, and (2) a method to measure and integrate the 3D information of the excavating point to the image. In addition, we conduct experiments using a real hydraulic excavator, and we demonstrate that an image with sufficient accuracy can be presented in real-time.

Autonomous Adaptive Flight Control of a UAV for Practical Bridge Inspection Using Multiple-Camera Image Coupling Method

Recently, with the deterioration of bridge facilities, demand has arisen for a method to inspect many bridges efficiently. One proposed bridge inspection method involves observation and inspection of cracks on undersides of bridges using a video camera mounted on an unmanned aerial vehicle (UAV) that flies under the bridges. There is an option to have a pilot operate the UAV, but it is desirable to have the UAV fly autonomously when efficiency of inspection is considered. Though there is a method using GPS for autonomous flight control of UAVs, there are many cases in which GPS cannot be utilized under bridges, and a new method is required for autonomous flight control in such places. The authors have already shown that autonomous flight control of UAVs can be achieved within the range of a monocular camera image by measuring the position of a UAV using camera images. However, since the flight range is bounded by the monocular camera image, it is necessary to move the camera position to fly the UAV autonomously in a wider space. In this paper, it is shown that a UAV can achieve autonomous flight control in wider spaces by constructing a single coordinate system for a combination of two camera images. In addition, considering that various measuring instruments might be mounted on a UAV, an adaptive control method capable of obtaining good control performance without changing the design parameters of the controllers should be applied. This method is useful for maintaining control performance when the total weight of the UAV changes. To show the effectiveness of our proposed method, we give an appropriate practical flight target orbit and present its experimental results.

Automatic Multi-Camera Extrinsic Parameter Calibration Based on Pedestrian Torsors.

Extrinsic camera calibration is essential for any computer vision task in a camera network. Typically, researchers place a calibration object in the scene to calibrate all the cameras in a camera network. However, when installing cameras in the field, this approach can be costly and impractical, especially when recalibration is needed. This paper proposes a novel, accurate and fully automatic extrinsic calibration framework for camera networks with partially overlapping views. The proposed method considers the pedestrians in the observed scene as the calibration objects and analyzes the pedestrian tracks to obtain extrinsic parameters. Compared to the state of the art, the new method is fully automatic and robust in various environments. Our method detect human poses in the camera images and then models walking persons as vertical sticks. We apply a brute-force method to determines the correspondence between persons in multiple camera images. This information along with 3D estimated locations of the top and the bottom of the pedestrians are then used to compute the extrinsic calibration matrices. We also propose a novel method to calibrate the camera network by only using the top and centerline of the person when the bottom of the person is not available in heavily occluded scenes. We verified the robustness of the method in different camera setups and for both single and multiple walking people. The results show that the triangulation error of a few centimeters can be obtained. Typically, it requires less than one minute of observing the walking people to reach this accuracy in controlled environments. It also just takes a few minutes to collect enough data for the calibration in uncontrolled environments. Our proposed method can perform well in various situations such as multi-person, occlusions, or even at real intersections on the street.

DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila.

Studying how neural circuits orchestrate limbed behaviors requires the precise measurement of the positions of each appendage in three-dimensional (3D) space. Deep neural networks can estimate two-dimensional (2D) pose in freely behaving and tethered animals. However, the unique challenges associated with transforming these 2D measurements into reliable and precise 3D poses have not been addressed for small animals including the fly, Drosophila melanogaster. Here, we present DeepFly3D, a software that infers the 3D pose of tethered, adult Drosophila using multiple camera images. DeepFly3D does not require manual calibration, uses pictorial structures to automatically detect and correct pose estimation errors, and uses active learning to iteratively improve performance. We demonstrate more accurate unsupervised behavioral embedding using 3D joint angles rather than commonly used 2D pose data. Thus, DeepFly3D enables the automated acquisition of Drosophila behavioral measurements at an unprecedented level of detail for a variety of biological applications.

BİR ENDÜSTRİYEL ROBOTUN KAFESSİZ ÇALIŞMASINI SAĞLAYACAK GÖRÜNTÜ TABANLI GÜVENLİK SİSTEMİNİN GELİŞTİRİLMESİ

İnsan ve robotun birlikte çalışması ile yaratılan işbirliği sayesinde, üretimde esneklik, verimlilik, kapasite ve kalite arttırılabilmektedir. Bu birlikte çalışma formunda (insan-robot etkileşiminde) insan ve robotun aynı ortamı paylaşması; çarpma/çarpışma olasılığını ortaya çıkardığından, bu türden çalışma ortamlarında güvenliği sağlamak en önemli sorun olmaktadır. Geleneksel robotlu sistemlerde güvenlik; kafes sistemleri, lazer tarayıcılar, ışık bariyerleri gibi çeşitli donanımlarla sağlanmaktadır. Ancak bu sistemler çalışma ortamını kısıtlamakta, insan ile robotun birlikte çalışmasında avantaj olarak öne sürülen esneklik faktörünü ortadan kaldırmaktadır. Bu çalışmanın ana amacı, kafes ve diğer önleyici sistemleri ortadan kaldırabilmek için robot üzerine konumlandırılan kamera ve donanımların kullanıldığı görüntü işleme tabanlı bir sistem ve algoritma ortaya çıkarabilmektir. Kullanılan kamera ve donanımların ihtiyaçları sağlayacak nitelikte, ancak düşük maliyetli standart donanımlardan seçilmesine dikkat edilmiştir. Böylelikle standart endüstriyel robotların etkileşime uygun formda çalıştırılabilmeleri için düşük maliyetli ve kolay uygulanabilir bir çözüm yaratılmaya çalışılmıştır. Bu amaçla; çalışmada, öncelikle çoklu kamera konumlandırılması ve görüntü elde edilmesi üzerine geliştirilen yöntem açıklanmıştır. Ardından elde edilmiş çoklu görüntülerden tekil bir panoramik görüntü elde edilmesi ve bu veri üzerinden insan/nesne tespiti ile konum ve mesafe çıkarımının nasıl yapıldığı belirtilmiştir. Bulunan parametreler ve tanımlı güvenlik kurallarını kullanarak insan ve robotun aynı ortamda çalışmasını düzenleyen algoritma özetlenmiş, son olarak da geliştirilen sistemin getirmiş olduğu kazanım ele alınmış ve ileride yapılabilecek çalışmalara ilişkin öngörüler verilmiştir.

Compressed Hadamard microscopy for high-speed optically sectioned neuronal activity recordings

Structured illumination microscopies achieve optical sectioning via differential modulation of in-focus and out-of-focus contributions to an image. Multiple wide-field camera images are analyzed to recreate an optical section. The requirement for multiple camera frames per image entails a loss of temporal resolution compared to conventional wide-field imaging. Here we describe a computational structured illumination imaging scheme, compressed Hadamard imaging, which achieves simultaneously high spatial and temporal resolution for optical sectioning of 3D samples with low-rank dynamics (e.g. neurons labeled with fluorescent activity reporters). We validate the technique with numerical simulations, and then illustrate with wide-area optically sectioned recordings of membrane voltage dynamics in mouse neurons in an acute brain slice and of calcium dynamics in zebrafish brain in vivo.

Cut-in vehicle warning system exploiting multiple rotational images of SVM cameras

Smart cruise control (SCC) is one of the representative systems of advanced driver assistance systems (ADAS). Conventional SCC systems have a problem in that they cannot detect a cut-in vehicle in a short distance because the field of view (FOV) of their sensors for detecting vehicles ahead is limited. To solve this problem, this paper proposes a novel method using surround view monitoring (SVM) cameras which can observe the surroundings of a host vehicle without blind spots. To reduce the variation of projected appearance according to the relative position, this paper proposes to exploit multiple virtual camera images (rotational images). In each rotational image, vehicles are detected using a part-based method: tires are detected and then vehicles are detected by combining the tires. Detected tires in multiple rotational images are integrated and tracked in 3D space. Finally, the proposed method determines whether a vehicle has intruded into the cut-in zone based on the position and orientation of the vehicle. The proposed method succeeds to detect vehicle even by a simple method thanks to the rotational image even without any special H/W. The proposed method uses only the sensor adopted in mass-produced vehicles and it is very practical. Moreover, the method can estimate the surrounding vehicle position accurately enough to be used in the braking control system. The performance of the proposed method is evaluated with various driving situations, of which the total length is 447.8 min (43.5 min for the test track and 404.3 min for natural driving). Among 211 cut-in events, 208 events are correctly warned and only 3 false warnings occur. Although the proposed method simultaneously processes the rotational images generated from three SVM cameras (forward, left, and right), the average processing time per frame takes about 58 ms.

실내 환경에서의 이동로봇의 위치추정을 위한 카메라 센서 네트워크 기반의 실내 위치 확인 시스템

This paper proposes a novel indoor positioning system (IPS) that uses a calibrated camera sensor network and dense 3D map information. The proposed IPS information is obtained by generating a bird’s-eye image from multiple camera images; thus, our proposed IPS can provide accurate position information when objects (e.g., the mobile robot or pedestrians) are detected from multiple camera views. We evaluate the proposed IPS in a real environment with moving objects in a wireless camera sensor network. The results demonstrate that the proposed IPS can provide accurate position information for moving objects. This can improve the localization performance for mobile robot operation.

Adaptive learning method for multiple-object detection in manufacturing

In manufacturing, vision-based inspections are effective nondestructive methods for implementing object discrimination. To minimize inspection times, multiple-object discrimination must be implemented for one inspection image. However, multiple-object inspections are difficult to perform in manufacturing because of the lack of distinct objects in inspection images. In addition, inspectors might fail to use multiple sensing devices when concurrently detecting objects. This article proposes a novel multiple-object sensing system that incorporates a local-adaptive-region-growing–based learning method for adaptively segmenting multiple-camera images for multiple-object discrimination. The proposed local-adaptive-region-growing method and support vector machine-based discrete wavelet transform can effectively classify multiple objects in the local subregions of inspection images. The proposed system bridges the gap between sensing devices and inspectors for solving problems encountered when concurrently using multiple sensing devices. The learning method yielded more favorable results than existing inspection methods have. Furthermore, a standard test series was developed for quantitatively comparing inspection methods.

어안 워핑 이미지 기반의 Ego motion을 이용한 위치 인식 알고리즘

This paper proposes a novel localization algorithm based on ego-motion which used Lucas-Kanade Optical Flow and warping image obtained through fish-eye lenses mounted on the robots. The omnidirectional image sensor is a desirable sensor for real-time view-based recognition of a robot because the all information around the robot can be obtained simultaneously. The preprocessing (distortion correction, image merge, etc.) of the omnidirectional image which obtained by camera using reflect in mirror or by connection of multiple camera images is essential because it is difficult to obtain information from the original image. The core of the proposed algorithm may be summarized as follows: First, we capture instantaneous <TEX>$360^{\circ}$</TEX> panoramic images around a robot through fish-eye lenses which are mounted in the bottom direction. Second, we extract motion vectors using Lucas-Kanade Optical Flow in preprocessed image. Third, we estimate the robot position and angle using ego-motion method which used direction of vector and vanishing point obtained by RANSAC. We confirmed the reliability of localization algorithm using ego-motion based on fisheye warping image through comparison between results (position and angle) of the experiment obtained using the proposed algorithm and results of the experiment measured from Global Vision Localization System.

Flow-Based Local Optimization for Image-to-Geometry Projection

The projection of a photographic data set on a 3D model is a robust and widely applicable way to acquire appearance information of an object. The first step of this procedure is the alignment of the images on the 3D model. While any reconstruction pipeline aims at avoiding misregistration by improving camera calibrations and geometry, in practice a perfect alignment cannot always be reached. Depending on the way multiple camera images are fused on the object surface, remaining misregistrations show up either as ghosting or as discontinuities at transitions from one camera view to another. In this paper we propose a method, based on the computation of Optical Flow between overlapping images, to correct the local misalignment by determining the necessary displacement. The goal is to correct the symptoms of misregistration, instead of searching for a globally consistent mapping, which might not exist. The method scales up well with the size of the data set (both photographic and geometric) and is quite independent of the characteristics of the 3D model (topology cleanliness, parametrization, density). The method is robust and can handle real world cases that have different characteristics: low level geometric details and images that lack enough features for global optimization or manual methods. It can be applied to different mapping strategies, such as texture or per-vertex attribute encoding.

Extrinsic analysis on manifolds is computationally faster than intrinsic analysis with applications to quality control by machine vision

In our technological era, non‐Euclidean data abound, especially because of advances in digital imaging. Patrangenaru (‘Asymptotic statistics on manifolds’, PhD Dissertation, 1998) introduced extrinsic and intrinsic means on manifolds, as location parameters for non‐Euclidean data. A large sample nonparametric theory of inference on manifolds was developed by Bhattacharya and Patrangenaru (J. Stat. Plann. Inferr., 108, 23–35, 2002; Ann. Statist., 31, 1–29, 2003; Ann. Statist., 33, 1211–1245, 2005). A flurry of papers in computer vision, statistical learning, pattern recognition, medical imaging, and other computational intensive applied areas using these concepts followed. While pursuing such location parameters in various instances of data analysis on manifolds, scientists are using intrinsic means, almost without exception. In this paper, we point out that there is no unique intrinsic analysis because the latter depends on the choice of the Riemannian metric on the manifold, and in dimension two or higher, there are infinitely such nonisometric choices. Also, using John Nash's celebrated isometric embedding theorem and an equivariant version, we show that for each intrinsic analysis there is an extrinsic counterpart that is computationally faster and give some concrete examples in shape and image analysis. The computational speed is important, especially in automated industrial processes. In this paper, we mention two potential applications in the industry and give a detailed presentation of one such application, for quality control in a manufacturing process via 3D projective shape analysis from multiple digital camera images. Copyright © 2011 John Wiley &amp; Sons, Ltd.

GFFD空間変形手法を用いたカメラ画像からの爪形状モデリング

This paper proposes a novel method of generating a 3D virtual foot model from multiple camera image data based on the GFFD deformation method. 3D virtual foot models for individual persons can be generated easily and rapidly with considerably low cost. By estimating the cause of errors that can occur in the whole process and endeavoring to lower the errors, we succeeded to achieve satisfactory accuracy. The errors are mainly classified into two kinds. One occurs in transforming process with use of the GFFD deformation method and the other in obtaining 3D coordinates of feature points. So, we devoted ourselves to eliminate the cause of both errors and repeated experiments again and again. As a result, we got to get a 3D virtual foot model within acceptable error range. The average error is 1.32∼1.72mm.

HMOAM: hybrid multi-objective genetic optimization for facial analysis by appearance model

Successful hybridization of single-objective evolutionary algorithm with gradient based methods has shown promising results. However, studies of hybridized Multi-Objective Evolutionary Algorithm are limited, especially in the domain of image analysis. This paper presents a novel methodology of hybridization of multi-objective genetic algorithm for the real world optimization problem of facial analysis of multiple camera images by 2.5D Appearance Model. Facial large lateral movements make acquisition and analysis of facial images by single camera inefficient. Moreover, non-convex multi-dimensional search space formed by the face search by appearance model requires an efficient optimization methodology. Currently, with wide availability of inexpensive cameras, a multi-view system is as practical as a single-view system. To manage these multiple informations, multi-objective genetic algorithm is employed to optimize the face search. To efficiently tackle the problem of non-convexity of the search space, hybridization of NSGA-II (Non-dominated Sorting Genetic Algorithm) with Gradient Descent is proposed in this paper. For this hybridization, we propose a gradient operator in NSGA-II, which computes gradients of the solutions in conjunction with the existing operator of mutation. Thus, it does not increase the computational cost of the system. Another proposition includes a unique method of calculating the relevant information of each camera in a multiple camera system which makes the hybridization procedure efficient and robust. Our proposed algorithm is applied on different facial poses of CMU-PIE database, webcam face images and synthetic face images, and the results are compared with a single view system and a non-hybrid multiple camera system. Simulation results validate the efficiency, accuracy and robustness achieved.

3D Virtual Foot Modeling from Multiple Camera Image Data Based on the GFFD Deformation Method

3D Virtual Foot Modeling from Multiple Camera Image Data Based on the GFFD Deformation Method

Multiple self-organizing maps for a visuo-motor system that uses multiple cameras with different fields of view

This article proposes multiple self-organizing maps (SOMs) for control of a visuo-motor system that consists of a redundant manipulator and multiple cameras in an unstructured environment. The maps control the manipulator so that it reaches its end-effector at targets given in the camera images. The maps also make the manipulator take obstacle-free poses. Multiple cameras are introduced to avoid occlusions, and multiple SOMs are introduced to deal with multiple camera images. Some simulation results are shown.

Objective quality assessment in free-viewpoint video production

This paper addresses the problem of objectively quantifying accuracy in free-viewpoint video production. Free-viewpoint video makes use of geometric scene reconstruction and renders novel views using the appearance sampled in multiple camera images. Previous work typically adopts an objective evaluation of geometric accuracy against ground-truth data or a subjective evaluation of visual quality in view synthesis. We consider two production scenarios, human performance capture in a highly constrained studio environment and sports production in a large-scale external environment. The accuracy of scene reconstruction is typically limited and absolute geometric accuracy does not necessarily reflect the quality of free-viewpoint rendering. A framework is introduced to quantify error at the point of view synthesis. The approach can be applied as a full-reference metric to measure fidelity to a ground-truth image or as a no-reference metric to measure the error in rendering. The framework is applied to a data set with known geometric accuracy and a comparison is presented for studio based and sports production scenarios.

Removal of adherent noise in images by using multiple cameras

AbstractIn this paper, we propose a new method for removing view‐disturbing noise from images taken with multiple cameras. In outdoor environments, scenes taken by cameras are often hard to see because of adherent noise on the surface of the lens‐protecting glass. The proposed method analyzes multiple camera images describing the same scene, and synthesizes an image in which adherent noise is eliminated. We show the effectiveness of the proposed method through experimental results. © 2008 Wiley Periodicals, Inc. Electr Eng Jpn, 164(3): 50–59, 2008; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/eej.20697

Human Foot Reconstruction from Multiple Camera Images with Foot Shape Database

Recently, researches and developments for measuring and modeling of the human body have been receiving much attention. Our aim is to reconstruct an accurate shape of a human foot from multiple camera images, which can capture dynamic behavior of the object. In this paper, a foot-shape database is used for accurate reconstruction of human foot. By using Principal Component Analysis, the foot shape can be represented with new meaningful variables. The dimensionality of the data is also reduced. Thus, the shape of object can be recovered efficiently, even though the object is partially occluded in some input views. To demonstrate the proposed method, two kinds of experiments are presented: reconstruction of human foot in a virtual reality environment with CG multi-camera images, and in real world with eight CCD cameras. In the experiments, the reconstructed shape error with our method is around 2 mm in average, while the error is more than 4 mm with conventional volume intersection method.