Real-world Coordinates Research Articles

Mobile Augmented Reality Application to Evaluate the River Flooding Impact in Coimbra

The downtown area of the city of Coimbra, Portugal, is at low altitude and has historically suffered floods that have caused serious economic losses. The present research proposes a mobile augmented reality (MAR) application aimed at visualising the effect of possible scenarios of flooding in an area of higher risk in the city. A realistic 3D model of the city was created, using data extracted with BLosm and processed through Blender, followed by its integration into Unity with Vuforia for AR visualisation. The methodology encompasses the extraction and simplification of 3D models, mapping real-world coordinates in Unity, analysing several datasets, obtaining a model through regression and implementing a workflow to manage interactions between various Unity objects. The MAR application enables users to visualise potential flood impacts on buildings, utilising colour-coded indicators to represent different levels of water contact. The system’s efficacy was evaluated by simulating various use-case scenarios, demonstrating the application’s capability to provide real-time, interactive flood risk assessments. The results underline the potential of integrating AR and machine learning for enhancing urban flood management and prevention.

A Virtual Reality Tool for Accuracy Assessment of 3D Models in an Immersive Virtual Environment

Abstract. Accurate validation and assessment techniques are essential for ensuring the reliability of spatial reconstructions derived from photogrammetry, enabling well-informed decision-making across diverse domains. This study presents a Virtual Reality (VR) based accuracy assessment tool tailored for evaluating the accuracy and quality of 3D models generated by Unmanned Aerial Vehicles (UAVs). Leveraging the Unity game engine platform, our workflow entails three key steps: aligning real-world coordinates with an arbitrary Unity coordinate system, transforming the positions of Ground Control Points (GCPs) from field survey to the arbitrary system using a reference GCP, and marking observed points on the 3D models. Absolute and Relative Root Mean Square Errors (RMSE), Mean Errors (ME), and Standard Deviation of errors (SD) are computed within the virtual environment via the game object transform properties. The error distributions around each GCP are visually depicted using Unity game engine components for enhanced interaction and comprehension. The efficacy of the tool is validated through experimentation on four 3D models generated from varying camera angles during UAV data capture. The tool provides the opportunity to directly interact with the 3D models and visualize the errors, which is quite distinct from traditional methods. Using the developed tool, results were obtained to indicate that configurations employing camera angles of 60° + 75º exhibit notable performance in terms of relative and absolute accuracy.

Predicting pedestrian-vehicle interaction severity at unsignalized intersections

Objectives This study aims to develop and validate a novel deep-learning model that predicts the severity of pedestrian-vehicle interactions at unsignalized intersections, distinctively integrating Transformer-based models with Multilayer Perceptrons (MLP). This approach leverages advanced feature analysis capabilities, offering a more direct and interpretable method than traditional models. Methods High-resolution optical cameras recorded detailed pedestrian and vehicle movements at study sites, with data processed to extract trajectories and convert them into real-world coordinates via precise georeferencing. Trained observers categorized interactions into safe passage, critical event, and conflict based on movement patterns, speeds, and accelerations. Fleiss Kappa statistic measured inter-rater agreement to ensure evaluator consistency. This study introduces a novel deep-learning model combining Transformer-based time series data capabilities with the classification strengths of a Multilayer Perceptron (MLP). Unlike traditional models, this approach focuses on feature analysis for greater interpretability. The model, trained on dynamic input variables from trajectory data, employs attention mechanisms to evaluate the significance of each input variable, offering deeper insights into factors influencing interaction severity. Results The model demonstrated high performance across different severity categories: safe interactions achieved a precision of 0.78, recall of 0.91, and F1-score of 0.84. In more severe categories like critical events and conflicts, precision and recall were even higher. Overall accuracy stood at 0.87, with both macro and weighted averages for precision, recall, and F1-score also at 0.87. The variable importance analysis, using attention scores from the proposed transformer model, identified ‘Vehicle Speed’ as the most significant input variable positively influencing severity. Conversely, ‘Approaching Angle’ and ‘Vehicle Distance from Conflict Point’ negatively impacted severity. Other significant factors included ‘Type of Vehicle’, ‘Pedestrian Speed’, and ‘Pedestrian Yaw Rate’, highlighting the complex interplay of behavioral and environmental factors in pedestrian-vehicle interactions. Conclusions This study introduces a deep-learning model that effectively predicts the severity of pedestrian-vehicle interactions at crosswalks, utilizing a Transformer-MLP hybrid architecture with high precision and recall across severity categories. Key factors influencing severity were identified, paving the way for further enhancements in real-time analysis and broader safety assessments in urban settings.

Utilizing Fisheye Cameras and Mass Spectrometry Imaging to Obtain Probing Coordinates for Histological Samples

Detecting chemical compositions in histological samples using mass spectrometry often requires time-consuming user interaction with a probe to ensure it is positioned as closely as possible to a sampling region. Fisheye cameras can obtain images of samples with a wide field-of-view; however, barrel distortion makes it difficult to analyze them as a flat plane. This research project aimed to develop an application that unwarps live images of histological samples captured with a fisheye camera and calculates the exact image-to-real-world probing coordinates with minor user interaction. Python’s PyQt5 and OpenCV libraries were utilized to create interactive software that removes fisheye distortion, finds image-to-real-world coordinates, and segments images. Connected hardware consisted of a Prusa 3D printer, liquid micro junction-surface sampling probe, fisheye camera, and LED light strips. The printer provided a bed for capturing sample images and control of the camera and probe within 1/100 of a millimetre. The software was outlined to process at least 2 photos of checkerboards for the fisheye undistortion, capture a photo of a sample to be unwarped, and require the user to map 4 points from the image to probe coordinates. Using this approach, real-world coordinates could be found for any image location. The application successfully unwarps live photos and converts image-to-real-world coordinates with sub-millimeter accuracy. Users can take new calibration photos, control lighting, save photos, and check undistortion entirely within the application to ensure all parameters are satisfactory. Coordinates of significant regions are displayed or may be manually selected, making it easier to sample specific areas. The success of unwarping relies on the camera’s focus and height, lighting, and object positioning. Future development plans to improve robustness and coordinate accuracy, and reduce the amount of calibration needed.

Automatic welding seam tracking and real-world coordinates identification with machine learning method

In the automatic welding process, the precision of the welding position significantly influences welding quality, relying heavily on the accurate identification of the welding seam. However, in complex and unstructured welding environments, such as those with strong arc lights, welding splashes, and thermal-induced deformations, various disturbances present a significant challenge to identify the welding seam. This paper introduces a cost-effective welding seam tracking model constructed using a laser line and a fixed viewpoint single RGB camera. The model's primary objective is to detect a chain of welding-seam points in both captured images and real-time camera feeds. The paper presents the techniques for camera calibration, laser detection, and the use of artificial intelligence to establish a real-world coordinate frame. The process begins with camera calibration using the OpenCV library to filter noise and undistort images. Subsequently, the thresholding technique is employed for color image segmentation, isolating the laser sensor light on the workpiece. An algorithm is then introduced to detect welding spots. Finally, the image coordinates are converted to world coordinates using two machine learning methods: Random Forest Regression and Gradient Boosting Regression. Experimental demonstrations were conducted at different positions on the object to assess algorithm precision. The results indicate average deviations of 0.83 mm, 0.74 mm, and 0.77 mm for X, Y, and Z coordinates, respectively, which fall below the expected standard deviation of 3 mm. Despite the overall consistency of the program, slight noise from environmental lighting may still exist. In conclusion, the proposed welding seam tracking method and coordinate conversion program have proven effective and cost-efficient, with potential for future improvements.

A computational account of transsaccadic attentional allocation based on visual gain fields

Coordination of goal-directed behavior depends on the brain's ability to recover the locations of relevant objects in the world. In humans, the visual system encodes the spatial organization of sensory inputs, but neurons in early visual areas map objects according to their retinal positions, rather than where they are in the world. How the brain computes world-referenced spatial information across eye movements has been widely researched and debated. Here, we tested whether shifts of covert attention are sufficiently precise in space and time to track an object's real-world location across eye movements. We found that observers' attentional selectivity is remarkably precise and is barely perturbed by the execution of saccades. Inspired by recent neurophysiological discoveries, we developed an observer model that rapidly estimates the real-world locations of objects and allocates attention within this reference frame. The model recapitulates the human data and provides a parsimonious explanation for previously reported phenomena in which observers allocate attention to task-irrelevant locations across eye movements. Our findings reveal that visual attention operates in real-world coordinates, which can be computed rapidly at the earliest stages of cortical processing.

Research on a Machine Vision-based Blade Measurement Method

A high-precision visual measurement method for the geometric dimensions of oil pump blades is proposed, utilizing an auxiliary measurement mechanism to assist in sub-pixel edge positioning. Based on images of calibration boards, the camera distortion is corrected by partitioning the image into regions and establishing a mapping index between pixel coordinates and real-world coordinates. Under fixed object and image distances, images of the oil pump blades and the contact between the blades and the pump are separately captured. The Canny operator and Sigmoid function are employed to fit the edges, extracting sub-pixel coordinates, which are then mapped to real-world coordinates. Finally, a least squares edge fitting is conducted to compute the dimensional parameters. Experimental results indicate that the use of auxiliary measurement mechanisms effectively enhances the precision of blade measurements, as compared to measurements solely based on oil pump blade images and images of the blade in contact with the pump.

River Ice Regime Recognition Based on Deep Learning: Ice Concentration, Area, and Velocity

The real-time derivation of the concentration, area, and velocity of river surface ice based on camera imagery is essential for predicting the potential risks related to ice blockages in water routes. The key lies in the continuous tracking and velocity measuring of river ice, and reliable ice motion detection is a prerequisite for the dynamic perception of tracking targets. Previous studies did not utilize motion tracking for measuring ice velocity, and particle image velocimetry and feature point matching were used. This study aimed to use deep learning methods to address the challenging problems of deriving the ice concentration, area, and velocity based on camera imagery, and the focus was on measuring the ice velocity and drawing trajectories using the particle video tracking algorithm. We built a dataset named IPC_RI_IDS and collected information during the ice cover break-up process in the Nenjiang River (China). Our suggested approach was divided into four steps: (1) image preprocessing, where the camera image was calibrated to real-world coordinates; (2) determining the ice and water pixels in the camera image using the lightweight semantic segmentation network and then calculating the ice concentration and area; (3) enhancing and optimizing motion detection using the semantic segmentation results; and (4) adapting the particle video tracking algorithm to measure ice velocity using the proposed tracking points generation strategy. Finally, we analyzed the surface ice data in the study area and attempted to predict the stage of the ice break-up process to provide support for the real-time short-term forecasts of ice floods.

An adaptive crack inspection method for building surface based on BIM, UAV and edge computing

Unmanned aerial vehicle (UAV) and building information modeling (BIM) have been applied to inspect building surfaces to improve efficiency and reduce labor costs. However, most of the current research focus on the inspection mode of “first collect and then identify” with limited intelligence. Therefore, this study proposes a “global-local” adaptive inspection method for building surfaces by integrating BIM, UAV (with multifunction camera), and edge computing, enabling real-time synchronization of crack data collection and analysis. First, the building surface area to be inspected is extracted from BIM to generate global inspection points using the selected camera's field of view (FOV) and considering coordinates transformation between BIM and reality. Then, A star algorithm integrating with genetic algorithm (GA) is used to optimize the global flight path based on generated inspection points. Finally, the coordinates of the inspection points based on BIM are converted into real-world coordinates, and the flight parameters such as the flight yaw angle (FYA) and the gimbal yaw angle (GYA) are calculated to generate the inspection mission. During global inspection, a local adaptive inspection is proposed for each FOV through real-time local crack identification and flight replanning using edge computing and zoom camera. In addition, multiple battery supply points are automatically added considering the endurance of the UAV. A real building has been used to validate the proposed “global-local” adaptive inspection method. The experiment results show that the inspection time has been significantly reduced compared with the traditional method, and the crack identification of building surfaces can be implemented with high efficiency, precision, quality, and low cost.

Multicamera edge-computing system for persons indoor location and tracking

This paper presents an indoor person localization and tracking system that uses multiple smart cameras equipped with artificial intelligence (AI) accelerators serving as edge-computing nodes. Our main contributions are as follows: (a) the development of a new multicamera tracking system for indoor scenarios; (b) the release of a multitarget multicamera tracking dataset; and (c) the development of an annotation mechanism based on waypoints. The system can simultaneously track several individuals while preserving their privacy and anonymity, because no images or sensitive data are transmitted outside the edge nodes. Only the position and appearance of each person were transmitted to the central server. In addition, a multitarget multicamera tracking dataset was released. The dataset contains recordings from five cameras in an indoor scenario and is annotated with the real-world coordinates of individuals. Ground-truth annotations were semiautomatically generated using a mechanism in which people equipped with mobile phones followed specific paths with predefined waypoints. Software related to the ground-truth annotation mechanism has also been released as open source.

Measuring food volume from RGB-Depth image with point cloud conversion method using geometrical approach and robust ellipsoid fitting algorithm

Volume measurement is an essential feature for food analysis and recognition systems to estimate nutritional intake from patients. Understanding the volume of consumed food is one approach for calculating how much food is consumed. This research provides a novel method for estimating food volume from RGB-Depth images using a combined point cloud conversion method and fitting points into ellipsoids using a geometrical approach. Point cloud segmentation is also utilized to separate the main object from the background. After the point cloud is downsampled, it is converted into real-world coordinates using the conversion method. The volume prediction is generated via numerical approaches, that is fitting to an ellipsoid using the least square approach. The objects in this study were tomatoes, oranges, kiwis, lemons, and potatoes, which were examined through multiple experiments. The evaluation results show that the predicted volume with the proposed method produces an Absolute Relative Error (ARE) of 2.6% for objects with a spherical shape, ARE of 3.1% for objects with an ellipsoid shape, and ARE of 3.0% for objects with an irregular surface, such as potatoes. When items are put on a white or blue background, volume predictions reach the ARE of 3.0% and 4.1%, respectively. As a result, white backgrounds were used more frequently in this study. Furthermore, experiments were carried out with various object orientations, camera distance locations, and object centers to evaluate the point cloud conversion method, yielding ARE of 3.4% and 4.4%, respectively. The obtained ARE indicates that the proposed approach makes reliable volume estimation.

Reproduction of spatial–temporal distribution of traffic loads on freeway bridges via fusion of camera video and ETC data

Understanding the spatial–temporal distribution of traffic loads on bridge decks is essential for managing traffic flow and assessing bridge load effects. While computer vision can already identify on-bridge vehicle parameters through traffic surveillance cameras, obtaining cost-effective vehicle load information remains a challenge, even though it is the most important service load for bridges. In this study, we present a methodology that applies computer vision algorithms to fuzzy match video-identified data with electronic toll collection (ETC) recorded data to reproduce the spatial–temporal distribution of traffic loads on freeway bridges. The proposed method involves three critical steps. First, multi-object detection and trajectory tracking of vehicle contours, vehicle heads, vehicle tires, and vehicle license plates are implemented using YOLOv5 and DeepSort algorithms. Second, the conversion of image coordinates to real-world coordinates is automatically calibrated using pre-measured roadside reflection points. Third, vehicle license plate characters are identified by the LPRnet algorithm and the results are fused with highway ETC records to determine vehicle weight information, while the KNN clustering algorithm is used to further infer vehicle axle weight. After these three steps, the vehicle load sequence in the video surveillance area can be reproduced. Finally, the effectiveness and feasibility of the proposed method are verified in an example of a highway bridge.

Scan2Drawing: Use of Deep Learning for As-Built Model Landscape Architecture

This paper presents an innovative and fully automatic solution of generating as-built computer-aided design (CAD) drawings for landscape architecture (LA) with three dimensional (3D) reality data scanned via drone, camera, and LiDAR. To start with the full pipeline, 2D feature images of ortho-image and elevation-map are converted from the reality data. A deep learning-based light convolutional encoder–decoder was developed, and compared with U-Net (a binary segmentation model), for image pixelwise segmentation to realize automatic site surface classification, object detection, and ground control point identification. Then, the proposed elevation clustering and segmentation algorithms can automatically extract contours for each instance from each surface or object category. Experimental results showed that the developed light model achieved comparable results with U-Net in landing pad segmentation with average intersection over union (IoU) of 0.900 versus 0.969. With the proposed data augmentation strategy, the light model had a testing pixel accuracy of 0.9764 and mean IoU of 0.8922 in the six-class segmentation testing task. Additionally, for surfaces with continuous elevation changes (i.e., ground), the developed algorithm created contours only have an average elevation difference of 1.68 cm compared to dense point clouds using drones and image-based reality data. For objects with discrete elevation changes (i.e., stair treads), the generated contours accurately represent objects’ elevations with zero difference using light detection and ranging (LiDAR) data. The contribution of this research is to develop algorithms that automatically transfer the scanned LA scenes to contours with real-world coordinates to create as-built computer-aided design (CAD) drawings, which can further assist building information modeling (BIM) model creation and inspect the scanned LA scenes with augmented reality. The optimized parameters for the developed algorithms are analyzed and recommended for future applications.

Non-destructive robotic sorting of cracked pistachio using deep learning

Pistachio, an agricultural product, is considered one of the guilt-free snacks due to its nutritional content, taste, and its health benefits. Usually, when it is consumed as a snack, in-shell pistachios are more desirable, whereas pistachios without their shell are ordinarily used for food preparation. One parameter important for pistachios sold as a snack is whether their shell is open. In pistachio processing plants, closed-shelled pistachios are separated with manual labor. There also exists an automated method called pin-picker machines, but as these machines harm pistachio kernels, they are undesirable. This paper offers a method to separate open-shelled pistachios from closed-shelled pistachios in an unharmful method. This method is done by using a deep learning-based object detection algorithm. In order to sort pistachios physically, a gripper and a conveyor system equipped Cartesian manipulator has been developed. In addition, a double-camera configuration is used. The key function of a double-camera configuration is to see the one-sided split on pistachios, which is hard to achieve with a single-camera configuration. Firstly, the object detection algorithm detects pistachios and gives their position with bounding boxes. Then their real-world coordinates are calculated, pistachios in both images taken with the cameras are paired, and the Cartesian manipulator separates pistachios. The detection accuracy was found as 98 % and 85 % for open-shelled pistachio and closed-shelled pistachio, respectively. With this method, closed-shelled pistachios can be sent to a cracking machine, or they can be sent to be stored for later use as they can withstand environmental effects better than open-shelled pistachios.

Autonomous Cricothyroid Membrane Detection and Manipulation Using Neural Networks and a Robot Arm for First-Aid Airway Management

Abstract Cricothyrotomy serves as one of the most efficient surgical interventions when a patient is enduring a can't intubate can't oxygenate (CICO) scenario. However, medical background and professional training are required for the provider to establish a patent airway successfully. Motivated by robotics applications in search and rescue, this work focuses on applying artificial intelligence techniques to the precise localization of the incision site, the cricothyroid membrane (CTM), of the injured using an RGB-D camera, and the manipulation of a robot arm with reinforcement learning to reach the detected CTM keypoint. In this paper, we proposed a deep learning-based model, the hybrid neural network (HNNet), to detect the CTM with a success rate of 96.6%, yielding an error of less than 5 mm in real-world coordinates. In addition, a separate neural network was trained to manipulate a robotic arm for reaching a waypoint with an error of less than 5 mm. An integrated system that combines both the perception and the control techniques was built and experimentally validated using a human-size manikin to prove the overall concept of autonomous cricothyrotomy with an RGB-D camera and a robotic manipulator using artificial intelligence.

3D printing on freeform surface: Real-time and accurate 3D dynamic dense surface reconstruction with HoloLens and displacement measurement sensors

This paper attempts to print irregular 3D objects on a freeform surface. Notably, it is inevitable to cause errors when creating a specific object shape in simulated and real-life scenarios. This is mainly attributed to the unique characteristics of the materials adopted, which tend to develop different deformations properties. Nevertheless, since the simulated model provides a clue regarding the coordinates of the deformed shape, it is adopted as an essential indicator in estimating accurate real-world coordinates. Concretely, this paper presents a complete system that is capable of reconstructing the detailed surface coordinates in real-time conditions. To verify the effectiveness of the proposed method, a metal board is used as the primary material to create different curvatures physically and virtually. In particular, the real-time performance of the overall 3D surface reconstruction has been experimentally evaluated using several tools, such as KUKA KR90 R3100 robots, HoloLens 2, displacement sensor, and ArUco markers. Consequently, both the quantitative and qualitative results are presented to demonstrate the feasibility of the proposed method. Furthermore, the experimental data obtained manifest that there are significant differences between the simulated and the real-world coordinates. Thus, the findings of this study provide an insightful outlook and lay down several important implications for future practice.

Indoor Localization With Robust Global Channel Charting: A Time-Distance-Based Approach

Fingerprinting (FP) significantly improves the indoor localization performance in non-line-of-sight-dominated areas. However, its deployment and maintenance is cost-intensive as it needs ground-truth reference systems for both the initial training and the adaption to environmental changes. Recently, channel charting (CC) has been investigated because it is an unsupervised method that does not need labeled measurement data. CC uses pairwise-distance metrics to estimate the physical distance between channel-state-information (CSI) measurements in the proximity. CC can then learn the underlying manifold of the channel measurements. While CC has shown promising results in modelling the local geometry of the radio environment, a deeper insight into CC for localization using multi-anchor large-bandwidth measurements is still pending. We therefore contribute a novel distance metric for CC that approaches a global linear correlation to the physical distance based on time-synchronized single-input/single-output (SISO) CSIs. This allows to learn the environment’s global geometry in a channel chart without annotations. We leverage a Siamese network, which enables CC-assisted FP only using a linear transformation from the chart to the real-world coordinates. We compare our approach to the state-of-the-art of CC on two different real-world data sets recorded with a 5G and UWB radio setup. Unlike FP-based localization which needs a large number of labeled data points to achieve a superior localization accuracy, our approach outperforms FP when only a few labeled data samples are available.

Towards a new method of estimating the student attention based on the eye gaze

This paper presents a new system for automating the monitoring and estimation of student attention during the course session. The followed approach is based on the analysis of the student's gaze to predict his state of attention. A simple hardware device consisting of a camera and a pc was used in this study. Existing machine learning algorithms were used for the student gaze estimation. The principles of homography were used to ensure the transformation from an image coordinates system to a real-world coordinates system. 5 students took part in this experiment and whose gaze was detected and analyzed during 10 minutes of the class session in order to analyze their states of attention and inattention.

Research on Autopilot Trajectory Planning Based on Deep Learning

Vehicle trajectory is usually used in driving behavior modeling and traffic safety. In order to obtain vehicle tracks in traffic video, an automatic acquisition method based on depth learning is proposed. In this paper, the YOLOV5 detector is used to detect vehicles in the region of interest in continuous frames, and the DEEPSORT tracker is used to track robust and fast vehicles. This tracker can effectively reduce the jump of target ID, improve tracking stability, obtain the real-world coordinates of vehicles through perspective transformation, and use a locally weighted regression algorithm to smooth the track. The experimental results show that the average multi-target tracking accuracy of the method in the expressway monitoring data set is 96.1%, and the average accuracy of trajectory acquisition is more than 93%. The smoothing algorithm can effectively remove the trajectory noise.

OCCLUSION SCREENING USING 3D CITY MODELS AS A REFERENCE DATABASE FOR MOBILE AR-APPLICATIONS

Abstract. Creating an immersive Augmented Reality (AR) experience requires aligning the digital content with the real environment so that the digital content appears and interacts in a similar way to a real object. For that, the virtual object must persist in its position across sessions and be occluded completely or partly, depending on whether a real-world object is in the line of sight. This research makes use of the cutting-edge technology available in the field of AR/MR, the Microsoft HoloLens 2.0, where the in-device Time of Flight (ToF) camera is used to scan the environment to create a Spatial Map. The CityGML LOD1 model enriches the Spatial Map and, in addition, is used as an occlusion mask via a custom rendering pipeline. Spatial Anchors can be tracked using the Simultaneous Localization And Mapping (SLAM) technique, used in conjunction with World Locking Tools to anchor the whole scene to the real-world coordinates. The developed occlusion effect is used in urban planning scenarios to introduce a new area design featuring a car-free environment. The proposed method is evaluated based on performance indicators, including frames per second (FPS) since it is highly correlated to user comfort. The findings of the users’ study demonstrate that the occlusion effect achieves its purpose since most of the participants reported enhancements in the depth perception and overall experience by enabling the occlusion screening.