Monocular Camera Research Articles

A low-cost indoor IoT object localization method based on monocular cameras at the edge device

A low-cost indoor IoT object localization method based on monocular cameras at the edge device

LDVI-SLAM: A Lightweight Monocular Visual-Inertial SLAM System for Dynamic Environments Based on Motion Constraints

Abstract Traditional Simultaneous Localization and Mapping (SLAM) systems are typically based on the assumption of a static environment. However, in practical applications, the presence of moving objects significantly reduces localization accuracy, limiting the system's versatility. To address the challenges of SLAM systems in dynamic environments, the academic community often employs computationally intensive methods such as deep learning, and some algorithms rely on expensive sensors (e.g., LiDAR or RGBD cameras) to obtain depth information. These factors increase computational complexity or hardware costs, complicating practical deployment. To improve localization accuracy and adaptability of SLAM systems in dynamic scenarios while maintaining low deployment costs, this paper proposes a dynamic environment robust monocular inertial SLAM system named LDVI-SLAM. The system uses more cost-effective sensors—monocular cameras and Inertial Measurement Unit (IMU)—along with lightweight computational methods. In LDVI-SLAM, first, the reliability of IMU data is verified. Then, using the ego-motion information provided by the IMU, along with epipolar constraint and an improved Rotation-aware Flow Vector Bound (R-FVB) constraint, dynamic feature points are eliminated. Additionally, this paper proposes a continuous tracking across interval frames method to enhance the distinction between static and dynamic feature points. Experimental results demonstrate that LDVI-SLAM performs effectively in dynamic environments and is easy to deploy. On the VIODE dataset, experimental results show that compared to the deep learning-based DynaSLAM, this method reduces the RMSE (Root Mean Square Error) of absolute trajectory error by 10.3%. Moreover, in terms of speed, under the same computing power, the single-frame processing speed of this method is comparable to ORB-SLAM3 and is two orders of magnitude faster than DynaSLAM, significantly outperforming deep learning-based SLAM algorithms. Experiments on the OMD dataset further prove that this method effectively avoids the risk of semantic classification errors, demonstrating better robustness and generality.

3D displacement measurement using a single-camera and mesh deformation neural network

Computer vision-based methods for civil structure’s vibration displacement measurement have emerged as useful tools in the recent years. These methods offer several benefits including non-contact measurements, cost-effectiveness, and the ability to capture full-field displacement. Yet, there remain certain challenges. Measuring vibration displacement in 3D typically requires multiple cameras, adding complexity to camera configurations. Moreover, existing methods relied heavily on physical markers or natural key points. Placing physical markers on structures is often impractical, and natural key points are difficult to detect on structures with few distinct features or during rapid movements. Contrary to previous approaches, this paper presents a novel technique that uses a monocular camera for 3D displacement measurements. This technique obviates the need for physical markers or the reliance on natural key points, representing a significant advancement. Central to the method is a deep neural network designed to predict 3D mesh deformation directly from a single image input, combined with an initial 3D cube mesh input. A synthetic 3D dataset is generated to train the neural network. In the testing phase for real structures, advanced video segmentation method is employed to remove the background in order to enhance the prediction accuracy. The practical efficacy of this methodology is validated in a laboratory through a series of experimental tests on beam structures, demonstrating reliable results and application potentials.

Automatic driving image matching via Random Sample Consensus (RANSAC) and Spectral Clustering (SC) with monocular camera.

In today's big data era, with the development of the Internet of Things (IoT) technology and the trend of autonomous driving prevailing, visual information has shown a blowout increase, but most image matching algorithms have problems such as low accuracy and low inlier rates, resulting in insufficient information. In order to solve the problem of low image matching accuracy and low inlier rate in the field of autonomous driving, this research innovatively applies spectral clustering (SC) in the field of data analysis to image matching in the field of autonomous driving, and a new image matching algorithm "SC-RANSAC" based on SC and Random Sample Consensus (RANSAC) is proposed. The datasets in this research are collected based on the monocular cameras of autonomous driving cars. We use RANSAC to obtain the initial inlier set and the SC algorithm to filter RANSAC's outliers and then use the filtered inliers as the final inlier set. In order to verify the effectiveness of the algorithm, it shows the matching effect from three angles: camera translation, rotation, and rotation and translation. SC-RANSAC is also compared with RANSAC, graph-cut RANSAC, and marginalizing sample consensus by using two different types of datasets. Finally, we select three representative pictures to test the robustness of the SC-RANSAC algorithm. The experimental results show that SC-RANSAC can effectively and reliably eliminate mismatches in the initial matching results; has a high inlier rate, real-time performance, and robustness; and can be effectively applied in the environment of autonomous driving.

Deep Learning-Based Detection and Robust Tracking of an Unmanned Surface Ship Using a Monocular Camera

Deep Learning-Based Detection and Robust Tracking of an Unmanned Surface Ship Using a Monocular Camera

A Novel Approach of Surface Texture Mapping for Cone-Beam Computed Tomography in Image-Guided Surgical Navigation.

The demand for cone-beam computed tomography (CBCT) imaging in clinics, particularly in dentistry, is rapidly increasing. Preoperative surgical planning is crucial to achieving desired treatment outcomes for imaging-guided surgical navigation. However, the lack of surface texture hinders effective communication between clinicians and patients, and the accuracy of superimposing a textured surface onto CBCT volume is limited by dissimilarity and registration based on facial features. To address these issues, this study presents a CBCT imaging system integrated with a monocular camera for reconstructing the texture surface by mapping it onto a 3D surface model created from CBCT images. The proposed method utilizes a geometric calibration tool for accurate mapping of the camera-visible surface with the mosaic texture. Additionally, a novel approach using 3D-2D feature mapping and surface parameterization technology is proposed for texture surface reconstruction. Experimental results, obtained from both real and simulation data, validate the effectiveness of the proposed approach with an error reduction to 0.32 mm and automated generation of integrated images. These findings demonstrate the robustness and high accuracy of our approach, improving the performance of texture mapping in CBCT imaging.

Identification and Positioning Method of Bulk Cargo Terminal Unloading Hopper Based on Monocular Vision Three-Dimensional Measurement

Rapid identification and localization of dry bulk cargo hoppers are currently core issues in the automation control of gantry cranes at dry bulk terminals. The current conventional method relies on LiDAR systems for the identification and positioning of bulk unloading hoppers. However, this approach is complex and costly. In contrast, GPS-based positioning solutions for bulk unloading hoppers are prone to damage due to the vibrations generated during the operation process. Therefore, in this paper, a hopper localization system based on monocular camera vision is proposed to locate the position of the bulk unloading hopper. The hopper identification and localization process are divided into three stages. The first stage uses the improved YOLOv5 model to quickly and roughly locate the hopper target. The second stage uses morphological geometrical features to locate the corner points of the hopper target. The third stage determines the three-dimensional coordinates of the hopper target by solving the position of the corner points in the world coordinate system through the PnP (Perspective-n-Point) algorithm. The experimental results show that the average positioning accuracy of the coordinates of the method is above 93%, demonstrating the accuracy and effectiveness of the method.

Vision-aided inertial navigation for planetary landing without feature extraction and matching

In this paper, a vision-aided inertial navigation method for planetary landing based on the direct method is proposed. This approach requires only a monocular camera and an inertial measurement unit (IMU) to be equipped on the lander. Unlike traditional feature-based visual-aided inertial navigation algorithms, which entail extracting and matching image features and thus incurring high computational costs, the proposed method utilizes the direct method that tracks any pixel with a sufficient intensity gradient. By jointly optimizing photometric and inertial errors, the proposed algorithm estimates the camera motion and sparse map of the scene. Simulation results demonstrate that the proposed method reduces computation by nearly 65 % compared to the feature point method, with a slight decrease in navigation accuracy.

Feature Detection of Non-Cooperative and Rotating Space Objects through Bayesian Optimization.

In this paper, we propose a Bayesian Optimization (BO)-based strategy using the Gaussian Process (GP) for feature detection of a known but non-cooperative space object by a chaser with a monocular camera and a single-beam LIDAR in a close-proximity operation. Specifically, the objective of the proposed Space Object Chaser-Resident Assessment Feature Tracking (SOCRAFT) algorithm is to determine the camera directional angles so that the maximum number of features within the camera range is detected while the chaser moves in a predefined orbit around the target. For the chaser-object spatial incentive, rewards are assigned to the chaser states from a combined model with two components: feature detection score and sinusoidal reward. To calculate the sinusoidal reward, estimated feature locations are required, which are predicted by Gaussian Process models. Another Gaussian Process model provides the reward distribution, which is then used by the Bayesian Optimization to determine the camera directional angles. Simulations are conducted in both 2D and 3D domains. The results demonstrate that SOCRAFT can generally detect the maximum number of features within the limited camera range and field of view.

A Multi-Plant Height Detection Method Based on Ruler-Free Monocular Computer Vision

Plant height is an important parameter of plant phenotype as one indicator of plant growth. In view of the complexity and scale limitation in current measurement systems, a scaleless method is proposed for the automatic measurement of plant height based on monocular computer vision. In this study, four peppers planted side by side were used as the measurement objects. Two color images of the measurement object were obtained by using a monocular camera at different shooting heights. Binary images were obtained as the images were processed by super-green grayscale and the Otsu method. The binarized images were transformed into horizontal one-dimensional data by the statistical number of vertical pixels, and the boundary points of multiple plants in the image were found and segmented into single-plant binarized images by filtering and searching for valleys. The pixel height was extracted from the segmented single plant image and the pixel displacement of the height was calculated, which was substituted into the calculation together with the reference height displacement to obtain the realistic height of the plant and complete the height measurements of multiple plants. Within the range of 2–3 m, under the light condition of 279 lx and 324 lx, this method can realize the rapid detection of multi-plant phenotypic parameters with a high precision and obtain more accurate plant height measurement results. The absolute error of plant height measurement is not more than ±10 mm, and the absolute proportion error is not more than ± 4%.

Latent Space Representations for Marker-Less Realtime Hand-Eye Calibration.

Marker-less hand-eye calibration permits the acquisition of an accurate transformation between an optical sensor and a robot in unstructured environments. Single monocular cameras, despite their low cost and modest computation requirements, present difficulties for this purpose due to their incomplete correspondence of projected coordinates. In this work, we introduce a hand-eye calibration procedure based on the rotation representations inferred by an augmented autoencoder neural network. Learning-based models that attempt to directly regress the spatial transform of objects such as the links of robotic manipulators perform poorly in the orientation domain, but this can be overcome through the analysis of the latent space vectors constructed in the autoencoding process. This technique is computationally inexpensive and can be run in real time in markedly varied lighting and occlusion conditions. To evaluate the procedure, we use a color-depth camera and perform a registration step between the predicted and the captured point clouds to measure translation and orientation errors and compare the results to a baseline based on traditional checkerboard markers.

Monocular Absolute Depth Estimation from Motion for Small Unmanned Aerial Vehicles by Geometry-Based Scale Recovery.

In recent years, there has been extensive research and application of unsupervised monocular depth estimation methods for intelligent vehicles. However, a major limitation of most existing approaches is their inability to predict absolute depth values in physical units, as they generally suffer from the scale problem. Furthermore, most research efforts have focused on ground vehicles, neglecting the potential application of these methods to unmanned aerial vehicles (UAVs). To address these gaps, this paper proposes a novel absolute depth estimation method specifically designed for flight scenes using a monocular vision sensor, in which a geometry-based scale recovery algorithm serves as a post-processing stage of relative depth estimation results with scale consistency. By exploiting the feature correspondence between successive images and using the pose data provided by equipped navigation sensors, the scale factor between relative and absolute scales is calculated according to a multi-view geometry model, and then absolute depth maps are generated by pixel-wise multiplication of relative depth maps with the scale factor. As a result, the unsupervised monocular depth estimation technology is extended from relative depth estimation in semi-structured scenes to absolute depth estimation in unstructured scenes. Experiments on the publicly available Mid-Air dataset and customized data demonstrate the effectiveness of our method in different cases and settings, as well as its robustness to navigation sensor noise. The proposed method only requires UAVs to be equipped with monocular camera and common navigation sensors, and the obtained absolute depth information can be directly used for downstream tasks, which is significant for this kind of vehicle that has rarely been explored in previous depth estimation studies.

Research on high-speed classification and location algorithm for logistics parcels based on a monocular camera

The rapid development of the logistics industry has driven innovations in parcel sorting technology, among which the swift and precise positioning and classification of parcels have become key to enhancing the performance of logistics systems. This study aims to address the limitations of traditional light curtain positioning methods in logistics sorting workshops amidst high-speed upgrades. This paper proposes a high-speed classification and location algorithm for logistics parcels utilizing a monocular camera. The algorithm combines traditional visual processing methods with an enhanced version of the lightweight YOLOv5 object detection algorithm, achieving high-speed, high-precision parcel positioning. Through the adjustment of the network structure and the incorporation of new feature extraction modules and ECIOU loss functions, the model’s robustness and detection accuracy have been significantly improved. Experimental results demonstrate that this model exhibits outstanding performance on a customized logistics parcel dataset, notably enhancing the model's parameter efficiency and computational speed, thereby offering an effective solution for industrial applications in high-speed logistics supply.

Depth-Enhanced Deep Learning Approach For Monocular Camera Based 3D Object Detection

Automatic 3D object detection using monocular cameras presents significant challenges in the context of autonomous driving. Precise labeling of 3D object scales requires accurate spatial information, which is difficult to obtain from a single image due to the inherent lack of depth information in monocular images, compared to LiDAR data. In this paper, we propose a novel approach to address this issue by enhancing deep neural networks with depth information for monocular 3D object detection. The proposed method comprises three key components: 1)Feature Enhancement Pyramid Module: We extend the conventional Feature Pyramid Networks (FPN) by introducing a feature enhancement pyramid network. This module fuses feature maps from the original pyramid and captures contextual correlations across multiple scales. To increase the connectivity between low-level and high-level features, additional pathways are incorporated. 2)Auxiliary Dense Depth Estimator: We introduce an auxiliary dense depth estimator that generates dense depth maps to enhance the spatial perception capabilities of the deep network model without adding computational burden. 3)Augmented Center Depth Regression: To aid center depth estimation, we employ additional bounding box vertex depth regression based on geometry. Our experimental results demonstrate the superiority of the proposed technique over existing competitive methods reported in the literature. The approach showcases remarkable performance improvements in monocular 3D object detection, making it a promising solution for autonomous driving applications.

A novel method for measuring center-axis velocity of unmanned aerial vehicles through synthetic motion blur images

Accurate velocity measurement of unmanned aerial vehicles (UAVs) is essential in various applications. Traditional vision-based methods rely heavily on visual features, which are often inadequate in low-light or feature-sparse environments. This study presents a novel approach to measure the axial velocity of UAVs using motion blur images captured by a UAV-mounted monocular camera. We introduce a motion blur model that synthesizes imaging from neighboring frames to enhance motion blur visibility. The synthesized blur frames are transformed into spectrograms using the Fast Fourier Transform (FFT) technique. We then apply a binarization process and the Radon transform to extract light-dark stripe spacing, which represents the motion blur length. This length is used to establish a model correlating motion blur with axial velocity, allowing precise velocity calculation. Field tests in a hydropower station penstock demonstrated an average velocity error of 0.048 m/s compared to ultra-wideband (UWB) measurements. The root-mean-square error was 0.025, with an average computational time of 42.3 ms and CPU load of 17%. These results confirm the stability and accuracy of our velocity estimation algorithm in challenging environments.

Enhancing Badminton Game Analysis: An Approach to Shot Refinement via a Fusion of Shuttlecock Tracking and Hit Detection from Monocular Camera.

Extracting the flight trajectory of the shuttlecock in a single turn in badminton games is important for automated sports analytics. This study proposes a novel method to extract shots in badminton games from a monocular camera. First, TrackNet, a deep neural network designed for tracking small objects, is used to extract the flight trajectory of the shuttlecock. Second, the YOLOv7 model is used to identify whether the player is swinging. As both TrackNet and YOLOv7 may have detection misses and false detections, this study proposes a shot refinement algorithm to obtain the correct hitting moment. By doing so, we can extract shots in rallies and classify the type of shots. Our proposed method achieves an accuracy of 89.7%, a recall rate of 91.3%, and an F1 rate of 90.5% in 69 matches, with 1582 rallies of the Badminton World Federation (BWF) match videos. This is a significant improvement compared to the use of TrackNet alone, which yields 58.8% accuracy, 93.6% recall, and 72.3% F1 score. Furthermore, the accuracy of shot type classification at three different thresholds is 72.1%, 65.4%, and 54.1%. These results are superior to those of TrackNet, demonstrating that our method effectively recognizes different shot types. The experimental results demonstrate the feasibility and validity of the proposed method.

A Robust Monocular and Binocular Visual Ranging Fusion Method Based on an Adaptive UKF.

Visual ranging technology holds great promise in various fields such as unmanned driving and robot navigation. However, complex dynamic environments pose significant challenges to its accuracy and robustness. Existing monocular visual ranging methods are susceptible to scale uncertainty, while binocular visual ranging is sensitive to changes in lighting and texture. To overcome the limitations of single visual ranging, this paper proposes a fusion method for monocular and binocular visual ranging based on an adaptive Unscented Kalman Filter (AUKF). The proposed method first utilizes a monocular camera to estimate the initial distance based on the pixel size, and then employs the triangulation principle with a binocular camera to obtain accurate depth. Building upon this foundation, a probabilistic fusion framework is constructed to dynamically fuse monocular and binocular ranging using the AUKF. The AUKF employs nonlinear recursive filtering to estimate the optimal distance and its uncertainty, and introduces an adaptive noise-adjustment mechanism to dynamically update the observation noise based on fusion residuals, thus suppressing outlier interference. Additionally, an adaptive fusion strategy based on depth hypothesis propagation is designed to autonomously adjust the noise prior of the AUKF by combining current environmental features and historical measurement information, further enhancing the algorithm's adaptability to complex scenes. To validate the effectiveness of the proposed method, comprehensive evaluations were conducted on large-scale public datasets such as KITTI and complex scene data collected in real-world scenarios. The quantitative results demonstrate that the fusion method significantly improves the overall accuracy and stability of visual ranging, reducing the average relative error within an 8 m range by 43.1% and 40.9% compared to monocular and binocular ranging, respectively. Compared to traditional methods, the proposed method significantly enhances ranging accuracy and exhibits stronger robustness against factors such as lighting changes and dynamic targets. The sensitivity analysis further confirmed the effectiveness of the AUKF framework and adaptive noise strategy. In summary, the proposed fusion method effectively combines the advantages of monocular and binocular vision, significantly expanding the application range of visual ranging technology in intelligent driving, robotics, and other fields while ensuring accuracy, robustness, and real-time performance.

Absolute velocity estimation of UAVs based on phase correlation and monocular vision in unknown GNSS‐denied environments

AbstractThis paper proposes a novel approach for absolute velocity estimation of unmanned aerial vehicles in unknown and unmapped GNSS‐denied environments. The proposed method leverages the advantages of Fourier‐based image phase correlation and utilizes off‐the‐shelf onboard sensors, including a downward‐facing monocular camera, an inertial sensor, and a sonar. The non‐matching tracking approach is particularly appealing, offering accurate estimation while remaining robust against frequency‐dependent noise, significant intensity variations, and time‐varying illumination disturbances. In the proposed method, the first step involves computing global pixel motion from consecutive images using phase correlation, which utilizes the shift property of the Fourier transform. This pixel motion estimation serves as the basis for creating a closed‐loop solution for absolute velocity estimation. To further enhance accuracy, a Kalman filter is implemented to fuse all available data and provide a reliable velocity estimate. Validation of the proposed visual‐inertial technique is conducted through simulation experiments using AirSim and real‐world flight tests. The results demonstrate the practicality and effectiveness of the approach across a range of challenging scenarios.

Smart insect-computer hybrid robots empowered with enhanced obstacle avoidance capabilities using onboard monocular camera

Insect-computer hybrid robots are receiving increasing attention as a potential alternative to small artificial robots due to their superior locomotion capabilities and low manufacturing costs. Controlling insect-computer hybrid robots to travel through terrain littered with complex obstacles of various shapes and sizes is still challenging. While insects can inherently deal with certain obstacles by using their antennae to detect and avoid obstacles, this ability is limited and can be interfered with by control signals when performing navigation tasks, ultimately leading to the robot being trapped in a specific place and having difficulty escaping. Hybrid robots need to add additional sensors to provide accurate perception and early warning of the external environment to avoid obstacles before getting trapped, ensuring smooth navigation tasks in rough terrain. However, due to insects’ tiny size and limited load capacity, hybrid robots are very limited in the sensors they can carry. A monocular camera is suitable for insect-computer hybrid robots because of its small size, low power consumption, and robust information acquisition capabilities. This paper proposes a navigation algorithm with an integrated obstacle avoidance module using a monocular camera for the insect-computer hybrid robot. The monocular cameras equipped with a monocular depth estimation algorithm based on deep learning can produce depth maps of environmental obstacles. The navigation algorithm generates control commands that can drive the hybrid robot away from obstacles according to the distribution of obstacle distances in the depth map. To ensure the performance of the monocular depth estimation model when applied to insect-computer hybrid robotics scenarios, we collected the first dataset from the viewpoint of a small robot for model training. In addition, we propose a simple but effective depth map processing method to obtain obstacle avoidance commands based on the weighted sum method. The success rate of the navigation experiment is significantly improved from 6.7% to 73.3%. Experimental results show that our navigation algorithm can detect obstacles in advance and guide the hybrid robots to avoid them before they get trapped.

A Robust Target Detection Algorithm Based on the Fusion of Frequency-Modulated Continuous Wave Radar and a Monocular Camera

Decision-level information fusion methods using radar and vision usually suffer from low target matching success rates and imprecise multi-target detection accuracy. Therefore, a robust target detection algorithm based on the fusion of frequency-modulated continuous wave (FMCW) radar and a monocular camera is proposed to address these issues in this paper. Firstly, a lane detection algorithm is used to process the image to obtain lane information. Then, two-dimensional fast Fourier transform (2D-FFT), constant false alarm rate (CFAR), and density-based spatial clustering of applications with noise (DBSCAN) are used to process the radar data. Furthermore, the YOLOv5 algorithm is used to process the image. In addition, the lane lines are utilized to filter out the interference targets from outside lanes. Finally, multi-sensor information fusion is performed for targets in the same lane. Experiments show that the balanced score of the proposed algorithm can reach 0.98, which indicates that it has low false and missed detections. Additionally, the balanced score is almost unchanged in different environments, proving that the algorithm is robust.