Inverse Perspective Transformation Research Articles

Deep Learning-Based Consistent Object Detection in Distance on Highly Perspective Region

CCTVs are commonly used for traffic monitoring and accident detection, but their images suffer from severe perspective distortion causing object size reduction with distance. This issue is exacerbated in tunnel CCTVs, positioned low due to space constraints, leading to challenging object detection, especially for distant small objects, due to perspective effects. To address this, this study proposes a solution involving a region of interest setup and an inverse perspective transformation technique. The transformed images, achieved through this technique, enlarge distant objects, maintaining object detection performance and appearance velocity across distances. To validate this, artificial CCTV images were generated in a virtual tunnel environment, creating original and transformed image datasets under identical conditions. Comparisons were made between the appearance velocity and object size of individual vehicles and for deep learning model performance with multiple moving vehicles. The evaluation was conducted across four distance intervals (50 m to 200 m) from the tunnel CCTV location. The results reveal that the model using original images experiences a significant decline in object detection performance beyond 100 m, while the transformed image-based model maintains a consistent performance up to the distance of 200 m.

Real-time risk assessment of road vehicles based on inverse perspective mapping

Pan/Tilt/Zoom (PTZ) cameras play an important role in traffic scenes due to their wide monitoring fields and high flexibility. However, since the focal length and angle of PTZ cameras change irregularly with the monitoring needs, it is difficult to obtain accurate physical information about the real world from the image information of PTZ cameras. Aiming to address the need for real-time risk assessment of road vehicles in traffic monitoring scenarios, a vehicle position and velocity measurement scheme based on camera inverse perspective transformation is proposed, along with a method for real-time risk assessment based on the position and velocity. Specifically, Firstly, the vehicle target in the video is detected and tracked by deep learning YOLO detection algorithm and optical flow tracking algorithm. According to the obtained trajectory set, the vanishing points in the road direction are calculated by Cascade Hough Transform and the road marking lines are detected. Then, according to the vanishing point and marking line, the camera calibration task is accomplished via exploratory focal length. After camera calibration, the camera-to-road inverse perspective transformation is applied to project the image plane onto the road surface and obtain, the actual position information of vehicles. Finally, the vehicle speed measurement and real-time road risk assessment are achieved by calculating the average of instantaneous velocities across multiple frames. Simulation experiment results in a traffic monitoring scenario demonstrate that this perspective-based method for real-time road vehicle risk assessment achieves good stability and practicality, which meets the requirements for vehicle speed measurement and real-time road risk assessment.

Deep learning–based intelligent measurement methods and system for CMM

Extensive manual intervention and management are typically required when using coordinate measuring machines (CMMs) for inspections in production lines leading to low efficiency. This study presents a deep learning–based intelligent measurement method and system for measuring typical features (including holes, cylinders, balls, steps, and slots) of common components to improve inspection efficiency. This method combines vision sensors and a trigger probe. The You Only Look Once algorithm was employed to learn and achieve intelligent detection of features. An image-matching algorithm based on image inverse perspective transformation was designed, and the ant colony algorithm was implemented to optimize the measurement sequence. Then, an automatic approach for feature measurement path planning was designed. The presented system was tested using CMM, and a component with multiple typical features was measured. Results show that this method and system can be efficaciously implemented for intelligent measurement of typical features.

Z-Increments Online Supervisory System Based on Machine Vision for Laser Solid Forming.

An improper Z-increment in laser solid forming can result in fluctuations in the off-focus amount during the manufacturing procedure, thereby exerting an influence on the precision and quality of the fabricated component. To solve this problem, this study proposes a closed-loop control system for a Z-increment based on machine vision monitoring. Real-time monitoring of the precise cladding height is accomplished by constructing a paraxial monitoring system, utilizing edge detection technology and an inverse perspective transformation model. This system enables the continuous assessment of the cladding height, which serves as a control signal for the regulation of the Z-increments in real-time. This ensures the maintenance of a constant off-focus amount throughout the manufacturing process. The experimental findings indicate that the proposed approach yields a maximum relative error of 1.664% in determining the cladding layer height, thereby enabling accurate detection of this parameter. Moreover, the real-time adjustment of the Z-increment quantities results in reduced standard deviations of individual cladding layer heights, and the height of the cladding layer increases. This proactive adjustment significantly enhances the stability of the manufacturing process and improves the utilization of powder material. This study can, therefore, provide effective guidance for process control and product optimization in laser solid forming.

Obstacle Detection Method Based on RSU and Vehicle Camera Fusion

Road obstacle detection is an important component of intelligent assisted driving technology. Existing obstacle detection methods ignore the important direction of generalized obstacle detection. This paper proposes an obstacle detection method based on the fusion of roadside units and vehicle mounted cameras and illustrates the feasibility of a combined monocular camera inertial measurement unit (IMU) and roadside unit (RSU) detection method. A generalized obstacle detection method based on vision IMU is combined with a roadside unit obstacle detection method based on a background difference method to achieve generalized obstacle classification while reducing the spatial complexity of the detection area. In the generalized obstacle recognition stage, a VIDAR (Vision-IMU based identification and ranging) -based generalized obstacle recognition method is proposed. The problem of the low accuracy of obstacle information acquisition in the driving environment where generalized obstacles exist is solved. For generalized obstacles that cannot be detected by the roadside unit, VIDAR obstacle detection is performed on the target generalized obstacles through the vehicle terminal camera, and the detection result information is transmitted to the roadside device terminal through the UDP (User Data Protocol) protocol to achieve obstacle recognition and pseudo-obstacle removal, thereby reducing the error recognition rate of generalized obstacles. In this paper, pseudo-obstacles, obstacles with a certain height less than the maximum passing height of the vehicle, and obstacles with a height greater than the maximum passing height of the vehicle are defined as generalized obstacles. Pseudo-obstacles refer to non-height objects that appear to be “patches” on the imaging interface obtained by visual sensors and obstacles with a height less than the maximum passing height of the vehicle. VIDAR is a vision-IMU-based detection and ranging method. IMU is used to obtain the distance and pose of the camera movement, and through the inverse perspective transformation, it can calculate the height of the object in the image. The VIDAR-based obstacle detection method, the roadside unit-based obstacle detection method, YOLOv5 (You Only Look Once version 5), and the method proposed in this paper were applied to outdoor comparison experiments. The results show that the accuracy of the method is improved by 2.3%, 17.4%, and 1.8%, respectively, compared with the other four methods. Compared with the roadside unit obstacle detection method, the speed of obstacle detection is improved by 1.1%. The experimental results show that the method can expand the detection range of road vehicles based on the vehicle obstacle detection method and can quickly and effectively eliminate false obstacle information on the road.

단안 카메라 기반 측위를 위한 터널 내 시설물의 거리 측정

The GNSS signal used for vehicle localization exhibits a shaded area such as a tunnel. To accurately estimate the location in the shaded area, the accumulated position error must be compensated with information obtained from a separate sensor. When a monocular camera is selected as an additional sensor, the process of measuring the relative distance after detecting objects in a tunnel should be performed before using these data as an indicator. This paper presents a modified inverse perspective transformation model that projects image pixels to the plane parallel to the ground. The image pixels are projected to the plane accounting for the distance and the height of the facility in the tunnel. Then, through the transformation model, the distance of the facility in the tunnel can be measured using a single camera sensor. The Samsungsan Tunnel located on National Highway 110 in South Korea was chosen for the experiment. The estimated distance results by the proposed method were compared with the LiDAR measurements obtained real-time time by computing the position errors for the longitudinal and lateral directions, separately. This study demonstrates the possibility of estimating a distance with only a monocular camera for precise positioning even in a tunnel.

The Extraction Method of Navigation Line for Cuttage and Film Covering Multi-Functional Machine for Low Tunnels

Aiming at the problem of low intelligence in the automatic navigation of the cuttage and film covering multi-functional machine for low tunnels, this study proposed a navigation line extraction method based on the improved YOLOv5s model, which can achieve the accurate extraction of navigation lines based on two planting methods of seedling transplanting and direct seeding. Firstly, we pre-processed the acquired images using inverse perspective transformation. Next, the Coordinate Attention and Ghost modules were applied to improve the YOLOv5s architecture, increasing the detection accuracy and speed of field targets. Finally, we extracted the feature points and fit the navigation lines based on the shape features of the targets using the geometric method. The experimental results showed that, compared with other algorithms, the accuracy of the proposed algorithm could reach more than 96%, the accuracy of navigation line extraction reached 98%, and the average detection time was 51 ms. The proposed method was robust and universal, and it can provide reliable navigation paths for the cuttage and film covering multi-functional machine.

Object ground lines regression and mapping from fisheye images to around view image for the AVP

The automated valet parking (AVP) based on the around view monitor (AVM) has attracted increasing attention from the research community due to its high practicality and low cost. However, the distortion correction and inverse perspective transformation of the around view images result in the truncated and deformed objects, which leads to poor detection performance. To address this problem, we propose a method of object ground lines regression and mapping from fisheye images to around view images. There are two key steps for the proposed method, regressing ground lines in the fisheye images and mapping ground lines to the around view image. To detect the objects and regress ground lines at the same time, we explored three new detectors based on YOLOv5 (i.e. YOLOv5-VGG, YOLOv5-Point, and YOLOv5-Ratio). Among them, YOLOv5-VGG adds a VGG-based regression network behind YOLOv5, YOLOv5-Point directly adds a regression head to share the convolutional features, and YOLOv5-Ratio combines the 2D box and ground line to predict its ratios. After that, the ground lines from four different views are mapped and fused to the around view image. Additionally, to facilitate the study of the object detection in the around view image, a large-scale labeled dataset is established, which comprises 9828 fisheye images collected from typical indoor and outdoor parking lots. For each image, the 2D bounding boxes and ground lines of objects are carefully labeled. Experiments show that the YOLOv5-Ratio obtains the best performance and can effectively detect the truncated and deformed object in the around view image with the precision rate of 92.28% and recall rate of 85.09% on our collected dataset.

A Topology Guided Method for Rail-Track Detection

Rail-track detection is a core function for automated rail transit perception. However, existing methods cannot effectively detect the rail-tracks in a complex environment, especially in turnout scenarios. In this study, we propose a topology guided method to detect rail-tracks which includes the following four parts: Firstly, a neural network is used to obtain the pixels of the rail-lanes, and the geometric relationship between rail-lanes is mined by inverse perspective transformation. Secondly, the rail-lanes’ pixels are converted to rail-lanes’ key points and the topological relationship between the key points. Thirdly, the rail-lanes are reconnected through topological relationships between key points. Finally, the rail-track geometry features are used to match the rail-lanes. Experimental results show that the rail-track level F1 score of the proposed method reached 91.62%, which is state-of-the-art (SOTA) in this field. Furthermore, the proposed method has been tested and applied on the Hong Kong Metro Tsuen Wan Line.

Perspective transformation based-initial value estimation for the speckle control points matching in an out-of-focus camera calibration using a synthetic speckle pattern.

Despite camera calibration methods using regular planar chessboard or circular marker array calibration targets having been widely used, the control point extraction accuracy is low if the image is defocused or if the noise level is high. Due to the noise robustness of digital image correlation (DIC) in speckle image matching, random speckle pattern is a better choice for camera calibration than chessboard or circular markers, if the imaging quality is low. The foremost process of this method is to conduct speckle control points matching DIC, where the initial value must be estimated close to the true value. It is challenging to provide accurate initial values for DIC if the difference of physical pixel scale is large between the reference image and the target image or if the target image is out-of-focus. To solve this problem, this work presents an efficient initial value estimation method for speckle control points matching using DIC, based on perspective transformation. Firstly, the four pairs of corners of the speckle regions in the reference image and target image are detected. Secondly, the target image is transformed to a new image that has the considerable size of pixel scale with the reference image, then four neighborhood points of the control point in the reference image and the corresponding points in the transformed new image are matched coarsely by fixed subset searching. Lastly, the matched points in the transformed target image are transformed back to the origin target image by the inverse perspective transformation matrix, then the initial value for DIC can be estimated by the matched four pairs of neighborhood points. Experiment results confirm the higher calibration accuracy delivered by the proposed method, rather than that of the chessboard or the circular marker array. Measurement precision is higher than the speckle pattern calibration method that uses SIFT-based initial value estimation.

Lane Image Detection Based on Convolution Neural Network Multi-Task Learning

Based on deep neural network multi-task learning technology, lane image detection is studied to improve the application level of driverless technology, improve assisted driving technology and reduce traffic accidents. The lane line database published by Caltech and Tucson company is used to extract the ROI (Region of Interest), scale, and inverse perspective transformation as well as to preprocess the image, so as to enrich the data set and improve the efficiency of the algorithm. In this study, ZFNet is used to replace the basic networks of VPGNet, and their structures are changed to improve the detection efficiency. Multi-label classification, grid box regression and object mask are used as three task modules to build a multi-task learning network named ZF-VPGNet. Considering that neural networks will be combined with embedded systems in the future, the network will be compressed to CZF-VPGNet without excessively affecting the accuracy. Experimental results show that the vision system of driverless technology in this study achieved good test results. In the case of fuzzy lane line and missing lane line mark, the improved algorithm can still detect and obtain the correct results, and achieves high accuracy and robustness. CZF-VPGNet can achieve high real-time performance (26FPS), and a single forward pass takes about 36 ms or less.

Semantic Segmentation of Parking Lot Scene Based on Surround View System

Semantic segmentation of parking lot scenes is the prerequisite of environment perception for automatic parking technology. It provides environmental semantic information for automatic parking of vehicles. However, due to the dim light of the parking lot environment, unclear ground signs, road reflections and other factors, semantic segmentation such as FCN The method is not yet able to segment ground signs such as background and lane lines to meet the perception needs of automatic parking. This paper proposes a parking lot scene semantic segmentation method based on a surround view system. The surround view system consists of 4 fisheye cameras. The images acquired by each camera are subjected to distortion correction, inverse perspective transformation and image stitching fusion to obtain a ring view. Based on the ring view, a semantic segmentation algorithm based on attention and feature fusion is proposed. Experiments are carried out on a self-made parking lot dataset with a size of 1280x960 pixels. The results show that the method proposed in this paper improves the mIoU of the FCN model by 12.3%.

Accurate and Lightweight RailNet for Real-Time Rail Line Detection

Railway transportation has always occupied an important position in daily life and social progress. In recent years, computer vision has made promising breakthroughs in intelligent transportation, providing new ideas for detecting rail lines. Yet the majority of rail line detection algorithms use traditional image processing to extract features, and their detection accuracy and instantaneity remain to be improved. This paper goes beyond the aforementioned limitations and proposes a rail line detection algorithm based on deep learning. First, an accurate and lightweight RailNet is designed, which takes full advantage of the powerful advanced semantic information extraction capabilities of deep convolutional neural networks to obtain high-level features of rail lines. The Segmentation Soul (SS) module is creatively added to the RailNet structure, which improves segmentation performance without any additional inference time. The Depth Wise Convolution (DWconv) is introduced in the RailNet to reduce the number of network parameters and eventually ensure real-time detection. Afterward, according to the binary segmentation maps of RailNet output, we propose the rail line fitting algorithm based on sliding window detection and apply the inverse perspective transformation. Thus the polynomial functions and curvature of the rail lines are calculated, and rail lines are identified in the original images. Furthermore, we collect a real-world rail lines dataset, named RAWRail. The proposed algorithm has been fully validated on the RAWRail dataset, running at 74 FPS, and the accuracy reaches 98.6%, which is superior to the current rail line detection algorithms and shows powerful potential in real applications.

Research on Detection Algorithm of Roadway Line on Structured Road Based on Vision

To solve the problem of low efficiency of lane recognition in different structured road environments, an improved Hough transform algorithm is proposed. Firstly, every frame of image in the video stream is processed by gray scale and inverse perspective transformation, so as to reduce the amount of data and improve the accuracy of lane recognition. Then, median filtering and wavelet hard threshold denoising are combined to remove noise and improve image quality. Secondly, the image is binarized and edge detected by OTSU algorithm and Sobel operator, and the lane feature information is extracted. Finally, the traditional Hough transform algorithm is improved, and the slopes and points of the selected straight lines in the feature area are averaged, and then a new straight line is obtained and displayed in the image. Simulation experiments show that the algorithm has good applicability and reliability in different structured road environments, and achieves the expected experimental results.

Design of the navigation system through the fusion of IMU and wheeled encoders

Mobile robots must autonomously navigate in structured environments. In this paper, we proposed a new navigation system for mobile robots, where lost-cost components were used to complete accurate navigation tasks. To accomplish the navigation task, we designed and validated a set of algorithms for obstacle avoidance and estimation of the robot pose. The proposed navigation system consists of three main modules: local occupancy map generation, extended Kalman filter (EKF)-based localization and modified obstacle avoidance algorithm. The system initially performed visual feature point tracking followed by feature point classification using an inverse perspective transformation for obstacle detection. When the estimated obstacle position is provided, a local obstacle avoidance algorithm is activated to generate motion commands for safe navigation. When the local occupancy map is generated, a modified artificial potential field based on boundary detection is developed for navigation through an indoor environment. Moreover, we proposed a strategy to localize a mobile robot using an IMU, a camera and wheel encoders. The EKF is adopted to reduce the sensor noise and bias. Experiments demonstrate that our navigation system is useful and can significantly benefit other reactive navigations.

Three-dimensional reconstruction filtering algorithm based on single ray picture of pipe welds

With constraints in single image, such as collinear equations and coplanarity, a real-scale object model can be obtained, and real scenes of regular objects can be reconstructed, by using three-dimensional reconstruction algorithms such as grid processing and interpolation fitting. A 3D reconstruction algorithm based on single pipe weld seam ray picture is proposed. Firstly, recognizing and segmenting the weld image, and using non-linear filtering to enhance the gray level of the image. Secondly, identifying the inner and outer edge positions of the weld, and performing threshold segmentation, morphological filtering, connected area detection, edge detection, line detection, image rotation, and inner wall area extraction. Thirdly, based on the image of the elliptical opening of the weld seam, the inverse perspective transformation is used to restore the formation of the weld seam root and the weld 3D scatter points. Finally, 3D point cloud is meshed, the function of point-to-surface is achieved, and a three-dimensional model of the welding tumor is generated. The experimental results demonstrate that the algorithm can accurately and effectively perform 3D reconstruction of the weld seam based on a single 2D pipe radiograph of the weld seam, and realize accurate identification of the weld nodules at the root of the pipe through-flow weld.

Global Positioning from a Single Image of a Rectangle in Conical Perspective.

This article presents a method to obtain the overall positioning of the focus of a camera from an image that includes a rectangle in a fixed reference with known position and dimension. This technique uses basic principles of descriptive geometry introduced in engineering courses. The document will first show how to obtain the dihedral projections of a rectangle after three turns and one translation. Secondly, we will proceed to obtain the image of the rectangle rotated in a conical perspective, taking the elevation plane as the drawing plane and a specific point in space as the view point, and represented in the dihedral system. Thirdly, we proceed with the inverse perspective transformation; we will expose a method to obtain the coordinates in the space of a rectangle obtained from an image. Finally, we check the method experimentally by taking an image of the rectangle with a camera in which the coordinates in the drawing plane (center of the image) are the only available position information. Then, the positioning and orientation of the camera in 3D will be obtained.

Table Recognition for Sensitive Data Perception in an IoT Vision Environment

Internet of Things (IoT) technology allows us to measure, compute, and decide about the physical world around us in a quantitative and intelligent way. It makes all kinds of intelligent IoT devices popular. We are continually perceived and recorded by intelligent IoT devices, especially vision devices such as cameras and mobile phones. However, a series of security issues have arisen in recent years. Sensitive data leakage is the most typical and harmful one. Whether we are just browsing files unintentionally in sight of high-definition (HD) security cameras, or internal ghosts are using mobile phones to photograph secret files, it causes sensitive data to be captured by intelligent IoT vision devices, resulting in irreparable damage. Although the risk of sensitive data diffusion can be reduced by optical character recognition (OCR)-based packet filtering, it is difficult to use it with sensitive data presented in table form. This is because table images captured by the intelligent IoT vision device face issues of perspective transformation, and interferences of circular stamps and irregular handwritten signatures. Therefore, a table-recognition algorithm based on a directional connected chain is proposed in this paper to solve the problem of identifying sensitive table data captured by intelligent IoT vision devices. First, a Directional Connected Chain (DCC) search algorithm is proposed for line detection. Then, valid line mergence and invalid line removal is performed for the searched DCCs to detect the table frame, to filter the irregular interferences. Finally, an inverse perspective transformation algorithm is used to restore the table after perspective transformation. Experiments show that our proposed algorithm can achieve accuracy of at least 92%, and filter stamp interference completely.

Lane Detection Algorithm for Intelligent Vehicles in Complex Road Conditions and Dynamic Environments.

Lane detection is an important foundation in the development of intelligent vehicles. To address problems such as low detection accuracy of traditional methods and poor real-time performance of deep learning-based methodologies, a lane detection algorithm for intelligent vehicles in complex road conditions and dynamic environments was proposed. Firstly, converting the distorted image and using the superposition threshold algorithm for edge detection, an aerial view of the lane was obtained via region of interest extraction and inverse perspective transformation. Secondly, the random sample consensus algorithm was adopted to fit the curves of lane lines based on the third-order B-spline curve model, and fitting evaluation and curvature radius calculation were then carried out on the curve. Lastly, by using the road driving video under complex road conditions and the Tusimple dataset, simulation test experiments for lane detection algorithm were performed. The experimental results show that the average detection accuracy based on road driving video reached 98.49%, and the average processing time reached 21.5 ms. The average detection accuracy based on the Tusimple dataset reached 98.42%, and the average processing time reached 22.2 ms. Compared with traditional methods and deep learning-based methodologies, this lane detection algorithm had excellent accuracy and real-time performance, a high detection efficiency and a strong anti-interference ability. The accurate recognition rate and average processing time were significantly improved. The proposed algorithm is crucial in promoting the technological level of intelligent vehicle driving assistance and conducive to the further improvement of the driving safety of intelligent vehicles.

An All-Weather Lane Detection System Based on Simulation Interaction Platform

Computer vision is an important part of the autonomous vehicles to gain the perception of the surrounding environment. In order to satisfy the adaptability of lane detection system under different illumination and different road conditions, an effective lane detection method based on image classification and hybrid isomeric operators is proposed. Camera correction matrix and distortion coefficient are obtained by using checkerboard grid images, and they will be cached to be shared with subsequent image streams to improve the speed of real-time lane detection. In the process of preprocessing, the images under different illumination conditions are classified and processed, and the inverse perspective transformation is used to deal with the 2-D image flow so that the image flow has a certain depth of data. According to the intensity of the whole gray mean, the edge detection is carried out by using heterogeneous operators and combining with a variety of threshold filtering methods to extract the lane pixels. The lane detection module transmits the vehicle steering angle and the deviate distance from the road center line to the decision layer, effectively implementing the interactive simulation. Experiments show that the algorithm has good real-time performance, stability, and robustness under different illumination and road conditions.