Homography Matrix Research Articles

UAV Low-Altitude Aerial Image Stitching Based on Semantic Segmentation and ORB Algorithm for Urban Traffic

UAVs are flexible in action, changeable in shooting angles, and complex and changeable in the shooting environment. Most of the existing stitching algorithms are suitable for images collected by UAVs in static environments, but the images are in fact being captured dynamically, especially in low-altitude flights. Considering that the great changes of the object position may cause the low-altitude aerial images to be affected by the moving foreground during stitching, so as to result in quality problems, such as splicing misalignment and tearing, a UAV aerial image stitching algorithm is proposed based on semantic segmentation and ORB. In the image registration, the algorithm introduces a semantic segmentation network to separate the foreground and background of the image and obtains the foreground semantic information. At the same time, it uses the quadtree decomposition idea and the classical ORB algorithm to extract feature points. By comparing the feature point information with the foreground semantic information, the foreground feature points can be deleted to realize feature point matching. Based on the accurate image registration, the image stitching and fusion will be achieved by the homography matrix and the weighted fusion algorithm. The proposed algorithm not only preserves the details of the original image, but also improves the four objective data points of information entropy, average gradient, peak signal-to-noise ratio and root mean square error. It can solve the problem of splicing misalignment tearing during background stitching caused by dynamic foreground and improves the stitching quality of UAV low-altitude aerial images.

Displacement Measurement Based on UAV Images Using SURF-Enhanced Camera Calibration Algorithm

Displacement is an important parameter in the assessment of the integrity of infrastructure; thus, its measurement is required in a multitude of guidelines or codes for structural health monitoring in most countries. To develop a low-cost and remote displacement measurement technique, a novel method based on an unmanned aerial vehicle (UAV) and digital image correlation (DIC) is presented in this study. First, an auxiliary reference image that meets the requirements is fabricated using the selected first image. Then, the speeded-up robust features (SURF) algorithm is introduced to track the feature points in the fixed areas. The least square algorithm is then employed to resolve the homography matrix of the auxiliary reference image and target images; then, the acquired homography matrices are utilized to calibrate the deviation caused by the UAV wobble. Finally, the integral pixel and sub-pixel matching of the DIC algorithm is employed to calculate the displacement of the target object. The numerical simulation results show that the proposed method has higher calculation accuracy and stability. The outdoor experiment results show that the proposed method has definite practicability.

Research on a Visual Servoing Control Method Based on Perspective Transformation under Spatial Constraint

Visual servoing has been widely employed in robotic control to increase the flexibility and precision of a robotic arm. When the end-effector of the robotic arm needs to be moved to a spatial point without a coordinate, the conventional visual servoing control method has difficulty performing the task. The present work describes space constraint challenges in a visual servoing system by introducing an assembly node and then presents a two-stage visual servoing control approach based on perspective transformation. A virtual image plane is constructed using a calibration-derived homography matrix. The assembly node, as well as other objects, are projected into the plane after that. Second, the controller drives the robotic arm by tracking the projections in the virtual image plane and adjusting the position and attitude of the workpiece accordingly. Three simple image features are combined into a composite image feature, and an active disturbance rejection controller (ADRC) is established to improve the robotic arm’s motion sensitivity. Real-time simulations and experiments employing a robotic vision system with an eye-to-hand configuration are used to validate the effectiveness of the presented method. The results show that the robotic arm can move the workpiece to the desired position without using coordinates.

Monitoring multiple geometrical dimensions in WAAM based on a multi-channel monocular visual sensor

Monitoring multiple geometrical dimensions is a crucial premise to realize precise control of layer size in wire and arc additive manufacturing (WAAM). Different from conventional single-target detection using a single sensor or multiple-target detection using multiple sensors, this study designs a novel multi-channel monocular vision sensor to monitor different geometrical sizes, i.e., current layer to nozzle distance, previous layer to nozzle distance, and layer width. A Marr-Hildreth algorithm is employed to extract boundaries of the molten pool and the previous layer. A homography matrix is solved for height calibration, and detected heights are introduced to assist layer width calibration. The sensor’s accuracy is verified via variable-size templates, and the maximum errors of detected height and width are 3.39% and 3.98%, respectively. Finally, the sensor is applied to monitor the layer size of a 5th-layered wall. This study lays a solid foundation for high-precision control of multiple geometrical sizes in WAAM.

Calibration of a multiview display with a tilted parallax barrier for reduction of nonlinear extrinsic crosstalk.

Multiview displays, which are capable of displaying different view-images for viewers at different positions, can suffer from undesirable view-image mixing, called crosstalk. In this paper, we propose a method to minimize a class of crosstalk that occurs due to misalignment of flat panel displays and parallax barriers that constitute multiview displays. More specifically, our method is aimed at cases where flat panel displays and parallax barriers are non-parallel. First, we introduce our framework, which associates homography matrices to sets of pixels visible through parallax barriers. Then, we propose an iterative method for finding pixels that are visible for a view-point. By using a stripe test pattern, the homography matrix that corresponds to a set of visible pixels can be found by monitoring the deformation of the stripe test pattern. Last, we experimentally check the validity of the iterative method and demonstrate that the proposed method successfully corrects the geometric distortion of a 3D image.

Motion detection in moving camera videos using background modeling and FlowNet

Real-time moving object detection is challenging for moving cameras due to the moving background. Many studies use homography matrix to compensate for global motion by warping the background model to the current frame. Then, the pixel difference between the current frame and the background model is used for background subtraction. Moving pixels are extracted by applying adaptive threshold and some post-processing techniques. On the other hand, deep learning-based dense optical flow can be efficient enough to extract the moving pixels, but it increases computational cost. This study proposes a method to enhance a classical background modeling method with deep learning-based dense optical flow. The main contribution of this paper is to propose a fusing algorithm for dense optical flow and background modeling approach. The background modeling methods are error-prone, especially with continuous camera movement, while the optical flow method alone may not always be efficient. Our hybrid method fuses both techniques to improve the detection accuracy. We propose a software architecture to run background modeling and dense optical flow methods in parallel processes. The proposed implementation approach significantly increases the method’s working speed, while the proposed fusion and combining strategy improve detection results. The experimental results show that the proposed method can run at high speed and has satisfying performance against the methods in the literature.

Localization of Cracks in Concrete Structures Using an Unmanned Aerial Vehicle.

Active research on crack detection technology for structures based on unmanned aerial vehicles (UAVs) has attracted considerable attention. Most of the existing research on localization of cracks using UAVs mounted the Global Positioning System (GPS)/Inertial Measurement Unit (IMU) on the UAVs to obtain location information. When such absolute position information is used, several studies confirmed that positioning errors of the UAVs were reflected and were in the order of a few meters. To address these limitations, in this study, without using the absolute position information, localization of cracks was defined using relative position between objects in UAV-captured images to significantly reduce the error level. Through aerial photography, a total of 97 images were acquired. Using the point cloud technique, image stitching, and homography matrix algorithm, 5 cracks and 3 reference objects were defined. Importantly, the comparative analysis of estimated relative position values and ground truth values through field measurement revealed that errors in the range 24–84 mm and 8–48 mm were obtained on the x- and y-directions, respectively. Also, RMSE errors of 37.95–91.24 mm were confirmed. In the future, the proposed methodology can be utilized for supplementing and improving the conventional methods for visual inspection of infrastructures and facilities.

Robust Detection Model of Vascular Landmarks for Retinal Image Registration: A Two-Stage Convolutional Neural Network.

Registration is useful for image processing in computer vision. It can be applied to retinal images and provide support for ophthalmologists in tracking disease progression and monitoring therapeutic responses. This study proposed a robust detection model of vascular landmarks to improve the performance of retinal image registration. The proposed model consists of a two-stage convolutional neural network, in which one segments the retinal vessels on a pair of images, and the other detects junction points from the vessel segmentation image. Information obtained from the model was utilized for the registration. The keypoints were extracted based on the acquired vascular landmark points, and the orientation features were calculated as descriptors. Then, the reference and sensed images were registered by matching keypoints using a homography matrix and random sample consensus algorithm. The proposed method was evaluated on five databases and seven evaluation metrics to verify both clinical effectiveness and robustness. The results established that the proposed method showed outstanding performance for registration compared with other state-of-the-art methods. In particular, the high and significantly improved registration results were identified on FIRE database with area under the curve (AUC) of 0.988, 0.511, and 0.803 in S, P, and A classes. Furthermore, the proposed method worked well on poor quality and multimodal datasets demonstrating an ability to achieve high AUC above 0.8.

Low-Altitude Photogrammetry and Remote Sensing in UAV for Improving Mapping Accuracy

The low-altitude photogrammetry technology of unmanned aerial vehicles (UAVs) is widely used in many fields, but the absence of analysis and research affects the accuracy of its data products. At the same time, low-altitude photogrammetry faces the problem of low elevation positioning accuracy. The network space triangulation adjustment in the beam technique region is considered to eliminate perspective distortion in non-overlapping areas. This paper explains the key technologies of low-altitude photography and remote sensing mapping of UAVs, rectifies the distortion difference of remote sensing images, and then carries out grid division on the image according to the improved APAP (as-projective-as-possible warp) matching method. Next, it solves each grid homography matrix, linearizes the homography matrix, and carries out image matching according to the linearized homography matrix, which can effectively weaken the ghosting phenomenon during image matching. The network space triangulation adjustment in the beam technique region is considered to eliminate perspective distortion in non-overlapping areas. The two measurement areas’ accuracy level is analyzed using digital line drawing and digital orthophoto images (DOIs). Finally, the experimental results indicate that the image matching algorithm proposed in this paper has strong reliability and can substantially increase photogrammetric elevation positioning.

Head and Eye Egocentric Gesture Recognition for Human-Robot Interaction Using Eyewear Cameras

Non-verbal communication plays a particularly important role in a wide range of scenarios in Human-Robot Interaction (HRI). Accordingly, this work addresses the problem of human gesture recognition. In particular, we focus on head and eye gestures, and adopt an egocentric (first-person) perspective using eyewear cameras. We argue that this egocentric view may offer a number of conceptual and technical benefits over scene- or robot-centric perspectives. A motion-based recognition approach is proposed, which operates at two temporal granularities. Locally, frame-to-frame homographies are estimated with a convolutional neural network (CNN). The output of this CNN is input to a long short-term memory (LSTM) to capture longer-term temporal visual relationships, which are relevant to characterize gestures. Regarding the configuration of the network architecture, one particularly interesting finding is that using the output of an internal layer of the homography CNN increases the recognition rate with respect to using the homography matrix itself. While this work focuses on action recognition, and no robot or user study has been conducted yet, the system has been designed to meet real-time constraints. The encouraging results suggest that the proposed egocentric perspective is viable, and this proof-of-concept work provides novel and useful contributions to the exciting area of HRI.

Про підхід до визначення відповідності між пікселями камери і проєктора у мультимедійному тирі

Предметом вивчення в статті є геометричні перетворення у системі«проєктор–екран–камера» для визначення відповідності між пікселямикамери і проєктора у мультимедійному тирі. Метою є розробка математичної моделі та алгоритму визначення відповідності між пікселямикамери і проєктора. Це потрібно для співставлення положення центроїдалазерної плями від пострілу в матриці камери і мішені, що генеруєтьсяу матриці проєктора. Завдання: формалізувати задачу виникнення геометричних спотворень у системі «проєктор–екран–камера»; розробитиматематичну модель визначення відповідності між пікселями камериі проєктора; вибрати ефективний алгоритм її вирішення. Використовуваними методами є: математична модель вирівнювання зображення наоснові гомографії, метод бінаризації з вибором значення порога методом Оцу; метод Дугласа–Пекера, який зменшує кількість точок, що апроксимують криву. Отримано такі результати. Згідно аналізу геометричних спотворень у системі «проєктор–екран–камера» сформульовано задачу розробки моделі для вирівнювання зображення з метоювизначення відповідності між пікселями камери і проєктора. Розроблено математичну модель і алгоритм вирівнювання точок зображення мішені у матриці камери з точками зображення мішені у матриці проєктора. Розроблено і програмно реалізовано алгоритм визначення відповідності між пікселями камери і проєктора. Математичнізалежності для визначення відповідності між пікселями камери і проєктора встановлено на основі матриці гомографії. Коефіцієнти матриціобчислюються за відповідними кутовими точками прямокутника матриці проєктора і його спотвореного трапецеїдального зображення наматриці камери. Розроблено алгоритм автоматичного визначення вершин трапеції та встановлення відповідності між пікселями камериі проєктора. Проведено аналіз та експериментальні дослідження факторів, що впливають на точність алгоритму: точність визначення вершин трапеції, за якими знаходяться коефіцієнти матриці гомографії; ступіньзаповнення області огляду камери зображенням із проєктора; відповідність роздільної здатності камери і проєктора. Дано рекомендації щодозменшення їхнього впливу. Зроблено такі висновки. Наукова новизнаотриманих результатів полягає в наступному: розроблено і експериментально досліджено математичну модель визначення відповідності міжпікселями камери і проєктора у мультимедійному тирі через вирівнюваннязображення, що відображається з екрана проєктора у камері, на основігомографії. До цього для вирівнювання зображень у мультимедійному тирі використовувалась модель на основі 2D-перетворень, у якій взаємнеположення у просторі проєктора, екрана і камери не враховується.Оскільки у гомографії враховуються тільки лінійні перетворення, у подальшому планується удосконалити модель — врахувати нелінійніспотворення, що виникають у об’єктивах камери і проєктора.

Ship Intention Prediction at Intersections Based on Vision and Bayesian Framework

Due to the high error frequency of the existing methods in identifying a ship’s navigational intention, accidents frequently occur at intersections. Therefore, it is urgent to improve the ability to perceive ship intention at intersections. In this paper, we propose an algorithm based on the fusion of image sequence and radar information to identify the navigation intention of ships at intersections. Some existing algorithms generally use the Automatic Identification System (AIS) to identify ship intentions but ignore the problems of AIS delay and data loss, resulting in unsatisfactory effectiveness and accuracy of intention recognition. Firstly, to obtain the relationship between radar and image, a cooperative target composed of a group of concentric circles and a central positioning radar angle reflector is designed. Secondly, the corresponding relationship of radar and image characteristic matrix is obtained after employing the RANSAC method to fit radar and image detection information; then, the homographic matrix is solved to realize radar and image data matching. Thirdly, the YOLOv5 detector is used to track the ship motion in the image sequence. The visual measurement model based on continuous object tracking is established to extract the ship motion parameters. Finally, the motion intention of the ship is predicted by integrating the extracted ship motion features with the position information of the shallow layer using a Bayesian framework. Many experiments on real data sets show that our proposed method is superior to the most advanced method for ship intention identification at intersections.

A calibration method for line-structured light system by using sinusoidal fringes and homography matrix

The existing calibration methods of line-structured light system have complicated calibration procedures. To address this issue, this paper presents a practicable calibration method based on a homography matrix. In this calibration method, it does not need the camera calibration as well as the light plane calibration. First, a sequence of the light-stripe images and sinusoidal fringe images are obtained at different positions moved by a translation stage. Second, a series of feature points located at the light plane will be extracted from these images. Then, a homography matrix, which is to map the relationship between the light plane and the image plane, can be calculated. The three-dimensional (3D) data of a tested object can be restored by this matrix after extracting the image coordinates of the light-stripe images. The experimental results have shown its feasibility and validity. Moreover, the entire calibration process is practicable to simplify the experimental procedures and easy to be applied in industrial inspection.

A marker-less monocular vision point positioning method for industrial manual operation environments

Vision-assisted technologies in industrial manual operation such as augmented reality (AR) are increasingly popular. They require high positioning accuracy and robustness to operate normally. However, narrow spaces, moving hands, or tools may occlude or obscure local visual features of operation environments and thus affect the positioning accuracy and robustness of operating position. The resultant misguidance may even cause misoperation of operators. This paper proposes a marker-less monocular vision point positioning method for vision-assisted manual operation in industrial environments. The proposed method can accurately and robustly locate the target point of operation using constraint minimization method even if the target area has no corresponding visual features in the case of occlusion and improper illumination. The proposed method has three phases: intersection generation, intersection optimization, and target point solving. In the intersection generation phase, a certain number of intersections of epipolar lines are generated as candidate target points using fundamental matrices. Here, the solving constraint is converted from point-to-line to point-to-points. In the intersection optimization phase, the intersections are optimized to two different sets through the iterative linear fitting and geometric mean absolute error methods. Here, the solving constraint is further converted from point-to-points to point-to-point sets. In the target point solving phase, the target point is solved as a constrained minimization problem based on the distribution constraint of the two intersection sets. Here, the solving constraint is finally converted from point-to-point sets to point-to-point, and the unique optimal solution is obtained as the target point. The experimental results show that this method has a better accuracy and robustness than the traditional homography matrix method for the practical industrial operation scenes.

Automatic RANSAC by Likelihood Maximization

In computer vision, and particularly in 3D reconstruction from images, it is customary to be faced with regression problems contaminated by outlying data. The standard and efficient method to deal with them is the Random Sample Consensus (RANSAC) algorithm. The procedure is simple and versatile, drawing random minimal samples from the data to estimate parameterized candidate models and ranking them based on the amount of compatible data. Such evaluation involves some threshold that separates inliers from outliers. In presence of unknown level of noise, as is usual in practice, it is desirable to remove the dependency on this fixed threshold and to estimate it as an additional unknown. Among the numerous variants of RANSAC, few, that we call 'automatic', propose this approach, which involves changing the maximization criterion of consensus, as it is naturally increasing with the varying threshold. An algorithm of Zach and Cohen (ICCV 2015) uses the likelihood statistics. We present the details and the implementation of their method along with quantitative and qualitative tests on standard stereovision tasks: estimation of homography, fundamental and essential matrix.

Cell-Based Target Localization and Tracking with an Active Camera

This paper proposes a new method of target localization and tracking. The method consists of four parts. The first part is to divide the scene into multiple cells based on the camera’s parameters and calibrate the position and error of each vertex. The second part mainly uses the bounding box detection algorithm, YOLOv4, based on deep learning to detect and recognize the scene image sequence and obtain the type, length, width, and position of the target to be tracked. The third part is to match each vertex of the cell in the image and the cell in the scene, generate a homography matrix, and then use the PnP model to calculate the precise world coordinates of the target in the image. In this process, a cell-based accuracy positioning method is proposed for the first time. The fourth part uses the proposed PTH model to convert the obtained world coordinates into P, T, and H values for the purpose of actively tracking and observing the target in the scene with a PTZ camera. The proposed method achieved precise target positioning and tracking in a 50 cm ∗ 250 cm horizontal channel and a vertical channel. The experimental results show that the method can accurately identify the target to be tracked in the scene, can actively track the moving target in the observation scene, and can obtain a clear image and accurate trajectory of the target. It is verified that the maximum positioning error of the proposed cell-based positioning method is 2.31 cm, and the average positioning error is 1.245 cm. The maximum error of the proposed tracking method based on the PTZ camera is 1.78 degrees, and the average error is 0.656 degrees.

Homography-Based Visual Servoing for Underactuated VTOL UAVs Tracking a 6-DOF Moving Ship

This paper develops a novel homography-based visual servo control method for the vertical take-off and landing (VTOL) unmanned aerial vehicle (UAV) tracking the trajectory of a 6 degrees of freedom (6-DOF) moving ship. Different from the classical homography-based visual servoing which is only suitable for the trajectory tracking of a planar moving target, an extended homography-based visual servoing framework is proposed to track the trajectory of a 6-DOF moving target regardless of its time-varying rotational motion. Taking entries in the homography matrix as feedback, the visual dynamics of UAV is established and decoupled into a translational motion sub-dynamics and an attitude sub-dynamics based on the hierarchical control strategy. In the translational motion subsystem, the robust controller is designed by embedding the nonlinear differentiator, the adaptive method, and a smooth saturation model in the backstepping control scheme. The saturation model is introduced to generate the constrained control inputs to ensure the nonsingular attitude extraction and help the visual points remain in the camera’s field of view (FoV) simultaneously. In the attitude subsystem, a robust state-constrained control method is proposed by utilizing a robust control barrier function (RCBF), the quadratic programming (QP) and a saturation model to guarantee the visibility, where RCBF can accommodate unknown dynamics. The theoretical analysis demonstrates the asymptotic stability of the closed-loop system. Simulations are carried out to further validate the controller performance.

Measuring Method of Slip Ratio for Tractor Driving Wheels Based on Machine Vision

Tractors are prone to large slips when they are in field operation. The degree of slip plays a vital role in traction efficiency and fuel efficiency. This paper presents a method for measuring the slip ratio of tractors in field operation based on machine vision. The accurate measurement of slip ratio needs to obtain actual velocity and theoretical velocity separately. For obtaining the actual velocity, a monocular camera mounted on the tractor vertically faces down at the ground to collect images. Then, the feature points of inter-frame ground images are matched by the ORB (Oriented FAST and Rotated BRIEF) algorithm for calculating the translational displacement. Next, a homography matrix based on camera calibration is proposed to complete the transformation of a point from the pixel coordinate system to the world coordinate system. Aiming to acquire the theoretical velocity, a method that takes the variations in tire radius into account is proposed, and the tire radii of the driving wheels are indirectly determined by the tire inflation pressure in real-time. The proposed measurement method was verified with an experimental tractor. The results show that the mean absolute errors of the tractor driving wheels’ slip ratio measured by the machine vision method are less than 0.75%, and the maximum of the absolute errors is not more than 2.22%, which shows good performance.

Fast Matching Method of UAV Aerial Photography Enhanced Low Illumination Image

Aiming at the problems of insufficient image contrast in three-dimensional reconstruction of UAV in low illumination environment and the unstable iteration times of the RANSAC algorithm in the feature matching process, real-time matching method of UAV aerial image is proposed. First, a new image enhancement algorithm is applied to the image to enhance its quality and visibility. Second, the enhanced fast algorithm in ORB extracts the feature points from the preprocessed image, and cross-matching performs rough matching. Finally, the PROSAC algorithm solves the homography matrix by selecting the highest quality interior points from the extracted feature points. To improve the matching accuracy, some exterior points that do not conform to the geometric characteristics of the image are removed based on the homography matrix and the set mismatch threshold. The results show that the improved ORB algorithm is applied to the low illumination environment of UAV aerial photography, the image matching accuracy in 3D reconstruction is improved, and the correct matching rate tends to 97.24~99.39%. The relevant research findings and conclusions provide a fast and effective method for UAV image matching in different low illumination environments.

Method for online matching of multiple identical targets using local homography matrix based on moving DLT

In the field of geometric measurement of stereo vision, problems such as interference sources and similarity of target characteristics increase the difficulty of the matching of multiple identical targets. An online matching method for multiple identical targets using local homography matrix based on moving direct linear transform is presented in this paper. Firstly, the matching feature point pairs between left and right background images are obtained in advance. Secondly, the left background image is evenly divided into multiple rectangular meshes, and the local homography matrix of each rectangular mesh is calculated offline. Thirdly, the mapping point of each target is estimated online by the local homography matrix corresponding to the rectangular mesh where the target is located, and the target point with the closest Euclidean distance to the mapping point is considered as the matching point. Computer simulation and quality experimental data proves effectiveness of the proposed method. Experimental results reflect that our method is better than existing matching methods based on Delaunay triangulation, and the root mean square error of image coordinates between the mapping points from our method and the actual matching points is less than 13 pixels when entire image size is 1920 pixels × 1200 pixels.