Mono Camera Research Articles

Curb Detection for a Pedestrian Assistance System using End-to-End Learning

Our goal is to develop an assistance system for supporting road crossing among older pedestrians. In order to accomplish this, we propose detecting the curb stone from the pedestrians’ point of view. Curb detection plays a significant role in road detection and obstacle avoidance, etc. However, it also presents significant challenges such as the small size of the target as well as, obstacles and different structures. To tackle these problems, we chose to fuse two sensors, a Camera and a Leddar, and use an algorithm that applies an end-to-end learning approach. The convolutional neural network was chosen to process the images acquired from the mono camera by filming the curb and its surroundings. The artificial neural network was selected to process the point cloud data of the Leddar acquired in the form of arrays from the 16 channels of the Leddar. A prototype was developed for data collection and testing purposes. It consists of a structure carrying both sensors mounted on a walker. The data from both sensors were collected with multiple factors taken into consideration, such as, weather, light conditions and, approaching angles. For the training of algorithms, an end-to-end learning approach was selected where we labelled the complete image or array rather than labelling the individual pixels or points in the data. The networks were trained and, the features from the parallel networks were concatenated and given as the input to the fully connected layers to train the complete network. The experimental results show an accuracy of more than 99% and robustness of the end-to-end learning approach. Both sensors are relatively inexpensive and are in fusion together, they are able to efficiently accomplish the task of detecting the curb stone from the pedestrians’ point of view.

Detection of Perpendicular Parking Spaces with a Mono Camera

Detection of Perpendicular Parking Spaces with a Mono Camera

DLRAD – A FIRST LOOK ON THE NEW VISION AND MAPPING BENCHMARK DATASET FOR AUTONOMOUS DRIVING

Abstract. DLRAD – a new vision and mapping benchmark dataset for autonomous driving is under development for the validation of intelligent driving algorithms. Stationary, mobile, and airborne sensors monitored simultaneously the environment around a reference vehicle, which was driving on urban, suburb and rural roads in and around the city of Braunschweig/Germany. Airborne images were acquired with the DLR 4k sensor system mounted on a helicopter. The DLR research car FASCarE is equipped with the latest sensor technology like front/rear radar, ultrasound and laser sensors, optical single and stereo cameras, and GNSS/IMU. Additionally, stationary terrestrial sensors like induction loops, optical mono and stereo cameras, radar and laser scanners monitor defined sections of the path from the ground. Simultaneously, the helicopter with the 4k sensor systems follows the reference car by keeping it all the time in the central nadir view. A next crucial step in the construction of the DLRAD benchmark dataset is the annotation of all objects in the reference dataset.The DLRAD benchmark dataset enables a huge variety of validation capabilities and opens a wide field of possibilities for the development, training and validation of machine learning algorithms in the context of autonomous driving. In this paper, we will present details of the sensor configurations and the acquisition campaign, which had taken place between the 18th July and 20th July 2017 in Braunschweig/Germany. Also, we show a first analysis of the data including the completeness and geometrical quality. The dataset will be published as soon as the coregistration and annotations are complete.

단안 카메라를 딥 러닝 네트워크 기반의 객체 검출 및 거리 추정

본 연구에서는 단안 카메라를 통한 실시간 차선 검출과 딥 러닝 네트워크를 기반으로 하는 객체 검출 및 거리 추정 시스템을 제안한다. 단안 카메라를 통해 취득한 이미지의 원근감을 제거한 뒤 Sliding Windows 기법을 이용해 차선에 해당하는 후보군을 선정하고, RANSAC 기법을 통해 차선을 검출한다. 객체 검출의 경우 기존의 Computer Vision 알고리즘보다 검출률이 높고 주변 환경 변화에 강인한 딥 러닝을 이용한다. 또한, 타 센서로부터 취득한 거리 데이터를 기반으로 단안 카메라에서 검출된 객체의 Bounding Box Pixel 값과 실제 거리 정보의 관계를 이용하여 수식을 세우고, 이를 통해 객체와 차량의 거리를 추정한다. 그리고 자율주행 차량의 안정적인 주행을 위해 종·횡 방향 제어를 한다. 그리고 차선유지 및 전방 객체와의 거리에 따른 위험도 판별을 긴급제동(Auto Emergency Braking) 실험을 통해 제안한 시스템의 성능을 검증한다.

METRIC SCALE CALCULATION FOR VISUAL MAPPING ALGORITHMS

Abstract. Visual SLAM algorithms allow localizing the camera by mapping its environment by a point cloud based on visual cues. To obtain the camera locations in a metric coordinate system, the metric scale of the point cloud has to be known. This contribution describes a method to calculate the metric scale for a point cloud of an indoor environment, like a parking garage, by fusing multiple individual scale values. The individual scale values are calculated from structures and objects with a-priori known metric extension, which can be identified in the unscaled point cloud. Extensions of building structures, like the driving lane or the room height, are derived from density peaks in the point distribution. The extension of objects, like traffic signs with a known metric size, are derived using projections of their detections in images onto the point cloud. The method is tested with synthetic image sequences of a drive with a front-looking mono camera through a virtual 3D model of a parking garage. It has been shown, that each individual scale value improves either the robustness of the fused scale value or reduces its error. The error of the fused scale is comparable to other recent works.

BAUM-1: A Spontaneous Audio-Visual Face Database of Affective and Mental States

In affective computing applications, access to labeled spontaneous affective data is essential for testing the designed algorithms under naturalistic and challenging conditions. Most databases available today are acted or do not contain audio data. We present a spontaneous audio-visual affective face database of affective and mental states. The video clips in the database are obtained by recording the subjects from the frontal view using a stereo camera and from the half-profile view using a mono camera. The subjects are first shown a sequence of images and short video clips, which are not only meticulously fashioned but also timed to evoke a set of emotions and mental states. Then, they express their ideas and feelings about the images and video clips they have watched in an unscripted and unguided way in Turkish. The target emotions, include the six basic ones (happiness, anger, sadness, disgust, fear, surprise) as well as boredom and contempt. We also target several mental states, which are unsure (including confused, undecided), thinking, concentrating, and bothered. Baseline experimental results on the BAUM-1 database show that recognition of affective and mental states under naturalistic conditions is quite challenging. The database is expected to enable further research on audio-visual affect and mental state recognition under close-to-real scenarios.

Development of A Multi-sensor Fusion-based Traffic Information Acquisition System with Robust to Environmental Changes using Mono Camera, Radar and Infrared Range Finder

Development of A Multi-sensor Fusion-based Traffic Information Acquisition System with Robust to Environmental Changes using Mono Camera, Radar and Infrared Range Finder

A Lane Following Mobile Robot Navigation System Using Mono Camera

In this paper, we develop a lane following mobile robot using mono camera. By using camera, robot can recognize its left and right side lane, and maintain the center line of robot track. We use Hough Transform for detecting lane, and PID controller for control direction of mobile robot. The validity of our robot system is performed in a real world robot track environment which is built up in our laboratory.

모바일 디바이스 기반의 멀티 모달 사용자 거리 추정 시스템

This paper present the multi-modal user distance estimation system using mono camera and mono microphone basically equipped with a mobile device. In case of a distance estimation method using an image, we is estimated a distance of the user through the skin color region extraction step, a noise removal step, the face and eyes region detection step. On the other hand, in case of a distance estimation method using speech, we calculates the absolute difference between the value of the sample of speech input. The largest peak value of the calculated difference value is selected and samples before and after the peak are specified as the ROI(Region of Interest). The samples specified perform FFT(Fast Fourier Transform) and calculate the magnitude of the frequency domain. Magnitude obtained is compared with the distance model to calculate the likelihood. We is estimated user distance by adding with weights in the sorted value. The result of an experiment using the multi-modal method shows more improved measurement value than that of single modality.

Autonomous Navigation for On-Orbit Servicing

On-orbit servicing missions induce challenges for the rendezvous and docking system since a typical target satellite is not specially prepared for such a mission, can be partly damaged or even freely tumbling with lost attitude control. In contrast to manned spaceflight or formation flying missions, new sensors and algorithms have to be designed for relative navigation. Dependent on the distance to the target, optical sensors such as mono and stereo cameras as well as 3D sensors like laser scanners can be employed as rendezvous sensors. Navigation methods for far and close range and different verification methods are discussed.

Finger Gesture-based Three-Dimension Mobile User Interaction Using a Rear-facing Camera

In this paper, we propose a 3D mobile user interaction for 3D space using a mono camera in mobile devices. The proposed system can support a UI using only the index finger of the hand holding the mobile device. Proposed fingertip detection algorithm was developed to overcome the limitation of the skin color segmentation of previous studies under varying lighting conditions and complex background using MMGC and AdaBoost algorithm. Also in proposed 3D mobile user interface, the estimated coordinates and area of the fingertip can be mapping 3D space of implemented application. The experimental result indicates the feasibility of the proposed algorithm for finger movement-based mobile user interfaces.

스케일-공간을 이용한 차선 마킹 후보 검출

Lane marking detection based on a mono camera sensor provides a low cost solution to both ITS (intelligent transportation systems) and DAS (driver assistant systems). A number of methods and implementations have been reported in the literature. However, reliable detection is still an issue. Traditional approaches are mostly based on statistics or Hough transforms. However, the former approaches usually include many irrelevant detection areas, and the latter are not practical because actual lanes are not usually suitable for the approximation with linear or polynomial equations. In this paper, we focus on increasing the reliability of detection by reducing the detection of irrelevant areas while improving the detection of actual lane marking areas, which is usually a tradeoff for most conventional approaches. We use scale-space for that. Through experiments with real images obtained from various environments, we could achieve a significant improvement over traditional approaches.

Vision Sensor-Based Driving Algorithm for Indoor Automatic Guided Vehicles

In this paper, we describe a vision sensor-based driving algorithm for indoor automatic guided vehicles (AGVs) that facilitates a path tracking task using two mono cameras for navigation. One camera is mounted on vehicle to observe the environment and to detect markers in front of the vehicle. The other camera is attached so the view is perpendicular to the floor, which compensates for the distance between the wheels and markers. The angle and distance from the center of the two wheels to the center of marker are also obtained using these two cameras. We propose five movement patterns for AGVs to guarantee smooth performance during path tracking: starting, moving straight, pre-turning, left/right turning, and stopping. This driving algorithm based on two vision sensors gives greater flexibility to AGVs, including easy layout change, autonomy, and even economy. The algorithm was validated in an experiment using a two-wheeled mobile robot.

Distance estimation using inertial sensor and vision

A simple distance estimation algorithm using inertial sensors and a mono camera is proposed. Two images of a target are obtained by moving a mono camera. The movement of the camera is estimated using inertial sensors and used as the baseline for the distance estimation. Through experiments, the accuracy of the proposed method is evaluated, where the distance error is 3.9% on average in a few meter ranges.

PARTICLE FILTERING FOR PEOPLE FOLLOWING BEHAVIOR USING LASER SCANS AND STEREO VISION

Mobile robots have a large application potential in everyday life. To build those applications some common and basic behaviors should be initially consolidated, including a people following behavior. In this paper a system able to follow a person based on information provided by a laser scan and a mono and stereo camera is presented. In order to accomplish this goal, a real-time particle filter system able to merge the information provided by the sensors (laser and 2D and 3D images) and calculate the position of the target is proposed, using probabilistic leg patterns, image features and optical flow to this end. The experiments carried out show promising results, allowing a real-time particle filtering based on two different information sources.

An Assistance System for Construction-Sites

The contribution describes an assistance system, which supports the driver with lateral functions in construction-sites. This system is based on short-range radar sensors for side observation. The observation ahead of the vehicle is based on 77Ghz long range radar and a mono camera. From the camera images, a 3D reconstruction is computed. The result is fused with data of the imaging radar sensor ARS300 using a so-called “Occupancy Grid” approach. Based on this, road boundaries are determined. The contribution shows further how road boundaries can be determined from this grid information. Together with dynamic objects and roadway markings, a current right and left delimitation of the allowed driving corridor is determined and a desired path for the ego vehicle is computed. For the lateral support, a lane centring function is combined with a loose lateral guidance called “virtual wall”. The “virtual wall” function is described in detail and the resulting performance in a test construction-site with truck dummy vehicle is shown.

Indoor Navigation and Localization for Visually impaired people Using Weighted Topological Map

Problem Statement: Image base methods are a new approach for solving problems of navigations for visually impaired people. Approach: The study introduced a new approach of an electronic cane for blind people using the environment represented as a weighted topological graph instead, each node contains images taken at some poses in the work space, instead of building a metric (3D) model of the environment. Results: By computing weights between already stored images and the real scene of the environment and take some considerations like sessions. The system gives advices for the blind person to select the right direction in indoor navigate depending on weights and session, where a mono camera cane-held gives information in front of the visually impaired person. Conclusion: A cane that has the ability of getting SIFT feature for an object or site from a sequence of live images using the suggested approach is very satisfactory, The session and weight, speed up the system and gives a wide range indoor navigation and may be used to outdoor. Experimental results demonstrated a good performance of proposed method, the identification of different scenes to the blind person done by constructing the weighted visual environment graph to the system. The proposed scheme is using SIFT features to represent the objects and the sites.

OPTICAL FLOW NAVIGATION FOR AN OUTDOOR UAV USING A WIDE ANGLE MONO CAMERA AND DEM MATCHING

Visual navigation for low altitude unmanned aerial vehicles (UAV) shows many scientific and technical challenges in terms of navigation performances, complex 3D environment, high computational requirements for real-time image processing and very limited onboard mass and power resources.The current paper presents a concept of a visual navigation system, based on a wide angle mono camera as vision sensor and matching of the recovered and reference 3D environment, modeled by a digital elevation map (DEM). The recovered 3D models are being produced by real-time optical flow processing of the navigation camera images. An embedded optical correlator is introduced, which allows a robust and ultra high-speed optical flow processing under different and even unfavorable illumination conditions. The proposed concept can be augmented by any state-of-the art navigation aids, e.g. GPS, IMU, magnetometers. The current paper discusses the general principles and structure of the navigation system as a vision only system, i.e. without any additional navigation aids, and with instantaneous determination of the vehicle state, i.e. without time filtering. The paper gives also detailed end-to-end visual navigation performance results, based on a detailed software simulation model of the visual navigation system, including the optical correlator as the key component for ultra-high speed image processing.

APPLICATION OF 3D-PMD VIDEO CAMERAS FOR TASKS IN THE AUTONOMOUS MOBILE ROBOTICS

Fast Localisation is a basic function within the mobile robotics. Beside the odometrie, mainly image sensors are used. This paper shows a short overview of the image sensors used for it. Outgoing from the 2D, 3D laser scanner, over 2D mono camera and 2D stereo camera, up to the omni-directional 2D camera are described problems, which appear in recording and modelling of the robot environment with such sensors. In this paper in comparison to the classical sensors a completely new 3D-PMD video camera is introduced as a more efficiently image sensor. This sensor can not only record the robot environment as normal 2D image but delivers furthermore additionally with the depth information into a three-dimensional video image. Concerning of speed and accuracy, new possibilities are opened in object and obstacle recognition, localization and autonomous map generation, within the mobile robotics.