Camera-based Methods Research Articles

Using 3D Hand Pose Data in Recognizing Human–Object Interaction and User Identification for Extended Reality Systems

Object detection and action/gesture recognition have become imperative in security and surveillance fields, finding extensive applications in everyday life. Advancement in such technologies will help in furthering cybersecurity and extended reality systems through the accurate identification of users and their interactions, which plays a pivotal role in the security management of an entity and providing an immersive experience. Essentially, it enables the identification of human–object interaction to track actions and behaviors along with user identification. Yet, it is performed by traditional camera-based methods with high difficulties and challenges since occlusion, different camera viewpoints, and background noise lead to significant appearance variation. Deep learning techniques also demand large and labeled datasets and a large amount of computational power. In this paper, a novel approach to the recognition of human–object interactions and the identification of interacting users is proposed, based on three-dimensional hand pose data from an egocentric camera view. A multistage approach that integrates object detection with interaction recognition and user identification using the data from hand joints and vertices is proposed. Our approach uses a statistical attribute-based model for feature extraction and representation. The proposed technique is tested on the HOI4D dataset using the XGBoost classifier, achieving an average F1-score of 81% for human–object interaction and an average F1-score of 80% for user identification, hence proving to be effective. This technique is mostly targeted for extended reality systems, as proper interaction recognition and users identification are the keys to keeping systems secure and personalized. Its relevance extends into cybersecurity, augmented reality, virtual reality, and human–robot interactions, offering a potent solution for security enhancement along with enhancing interactivity in such systems.

See the Unseen: Grid-Wise Drivable Area Detection Dataset and Network Using LiDAR

Drivable Area (DA) detection is crucial for autonomous driving. Camera-based methods rely heavily on illumination conditions and often fail to capture accurate 3D information, while LiDAR-based methods offer accurate 3D data and are less susceptible to illumination conditions. However, existing LiDAR-based methods focus on point-wise detection, so are prone to occlusion and limited by point cloud sparsity, which leads to decreased performance in motion planning and localization. We propose Argoverse-grid, a grid-wise DA detection dataset derived from Argoverse 1, comprising over 20K frames with fine-grained BEV DA labels across various scenarios. We also introduce Grid-DATrNet, a first grid-wise DA detection model utilizing global attention through transformers. Our experiments demonstrate the superiority of Grid-DATrNet over various methods, including both LiDAR and camera-based approaches, in detecting grid-wise DA on the proposed Argoverse-grid dataset. Grid-DATrNet achieves state-of-the-art results with an accuracy of 93.28% and an F1-score of 0.8328. We show that Grid-DATrNet can detect grids even in occluded and unmeasured areas by leveraging contextual and semantic information through global attention, unlike CNN-based DA detection methods. The preprocessing code for Argoverse-grid, experiment code, Grid-DATrNet implementation, and result visualization code are available at AVE Laboratory official git hub.

Monocular BEV Perception of Road Scenes via Front-to-Top View Projection.

HD map reconstruction is crucial for autonomous driving. LiDAR-based methods are limited due to expensive sensors and time-consuming computation. Camera-based methods usually need to perform road segmentation and view transformation separately, which often causes distortion and missing content. To push the limits of the technology, we present a novel framework that reconstructs a local map formed by road layout and vehicle occupancy in the bird's-eye view given a front-view monocular image only. We propose a front-to-top view projection (FTVP) module, which takes the constraint of cycle consistency between views into account and makes full use of their correlation to strengthen the view transformation and scene understanding. In addition, we apply multi-scale FTVP modules to propagate the rich spatial information of low-level features to mitigate spatial deviation of the predicted object location. Experiments on public benchmarks show that our method achieves various tasks on road layout estimation, vehicle occupancy estimation, and multi-class semantic estimation, at a performance level comparable to the state-of-the-arts, while maintaining superior efficiency.

Three-Dimensional Human Posture Recognition by Extremity Angle Estimation with Minimal IMU Sensor.

Recently, posture recognition technology has advanced rapidly. Herein, we present a novel posture angle calculation system utilizing a single inertial measurement unit and a spatial geometric equation to accurately identify the three-dimensional (3D) motion angles and postures of both the upper and lower limbs of the human body. This wearable system facilitates continuous monitoring of body movements without the spatial limitations or occlusion issues associated with camera-based methods. This posture-recognition system has many benefits. Providing precise posture change information helps users assess the accuracy of their movements, prevent sports injuries, and enhance sports performance. This system employs a single inertial sensor, coupled with a filtering mechanism, to calculate the sensor's trajectory and coordinates in 3D space. Subsequently, the spatial geometry equation devised herein accurately computed the joint angles for changing body postures. To validate its effectiveness, the joint angles estimated from the proposed system were compared with those from dual inertial sensors and image recognition technology. The joint angle discrepancies for this system were within 10° and 5° when compared with dual inertial sensors and image recognition technology, respectively. Such reliability and accuracy of the proposed angle estimation system make it a valuable reference for assessing joint angles.

A novel machine learning-based approach for in-situ surface roughness prediction in laser powder-bed fusion

Controlling and optimizing surface roughness remain a significant challenge in laser powder bed fusion (LPBF). Surface roughness affects printed part quality, particularly fatigue life, leading to costly post-processing. Nevertheless, tailored roughness can be beneficial to specific fields in engineering and medicine, such as cooling or osteointegration. Consequently, the importance of quality assurance, in-situ monitoring, and automatic anomaly detection has been elevated among the users of the LPBF process. In-situ surface roughness detection remains a less explored area within all in-situ defect detection. Additionally, existing studies primarily employ camera-based methods, which come with inherent limitations, including sensitivity to ambient light conditions, compromises between resolution and field of view, and the requirement for additional equipment such as adaptive optical filters. This study pioneers using a photodiode sensor, offering faster response times for real-time surface roughness prediction. Integrating this sensor with machine learning (ML) algorithms establishes a robust framework for surface roughness prediction. The methodology involves analyzing the captured light intensity from the melt pool by an on-axial photodiode and incorporating additional process variables into ML models to predict surface roughness within each small area of the printed part (690 µm × 510 µm), even at edges and corners. Multiple ML algorithms are trained and rigorously validated with unseen data. This comprehensive analysis encompasses the impact of process parameter variations on a wide range of surface roughness values, affirming the efficiency of the proposed methodology.

Occupancy Estimation from Blurred Video: A Multifaceted Approach with Privacy Consideration

Building occupancy information is significant for a variety of reasons, from allocation of resources in smart buildings to responding during emergency situations. As most people spend more than 90% of their time indoors, a comfortable indoor environment is crucial. To ensure comfort, traditional HVAC systems condition rooms assuming maximum occupancy, accounting for more than 50% of buildings’ energy budgets in the US. Occupancy level is a key factor in ensuring energy efficiency, as occupancy-controlled HVAC systems can reduce energy waste by conditioning rooms based on actual usage. Numerous studies have focused on developing occupancy estimation models leveraging existing sensors, with camera-based methods gaining popularity due to their high precision and widespread availability. However, the main concern with using cameras for occupancy estimation is the potential violation of occupants’ privacy. Unlike previous video-/image-based occupancy estimation methods, we addressed the issue of occupants’ privacy in this work by proposing and investigating both motion-based and motion-independent occupancy counting methods on intentionally blurred video frames. Our proposed approach included the development of a motion-based technique that inherently preserves privacy, as well as motion-independent techniques such as detection-based and density-estimation-based methods. To improve the accuracy of the motion-independent approaches, we utilized deblurring methods: an iterative statistical technique and a deep-learning-based method. Furthermore, we conducted an analysis of the privacy implications of our motion-independent occupancy counting system by comparing the original, blurred, and deblurred frames using different image quality assessment metrics. This analysis provided insights into the trade-off between occupancy estimation accuracy and the preservation of occupants’ visual privacy. The combination of iterative statistical deblurring and density estimation achieved a 16.29% counting error, outperforming our other proposed approaches while preserving occupants’ visual privacy to a certain extent. Our multifaceted approach aims to contribute to the field of occupancy estimation by proposing a solution that seeks to balance the trade-off between accuracy and privacy. While further research is needed to fully address this complex issue, our work provides insights and a step towards a more privacy-aware occupancy estimation system.

P‐44: A Smart Eye Tracking System Based on Linear Array Sensors

Eye tracking as an important human‐machine interface plays avital role in immersive displays such as augmented reality (AR) and virtual reality (VR). Different from near infrared (NIR) camerabased methods, the proposed corneal reflection method based on principle of eyeball co‐vision takes advantage of using eye‐safe 940‐ nm vertical cavity surface emitting laser (VCSEL) and dual‐column linear array sensor (DCLAS) to obtain coordinates of the eyeball's moving positions through detecting its reflected light. The DCLAS not only delivers high sensitivity at eye‐safe NIR wavelength, but also has high spatial resolution for position accuracy and wide dynamic range to suppress ambient light influence. Together with low power consumption, good wearability, low data redundancy and simple computing algorism, a great promise can be made to enable smart eye tracking for AR/VR.

A Digital Camera-Based Eye Movement Assessment Method for NeuroEye Examination

The ability to perform quantitative and automated neurological assessment could enhance diagnosis and treatment in the pre-hospital setting, such as during telemedicine or emergency medical services (EMS) encounters. Such a tool could be developed by adapting clinically significant information such as symmetry of eye movement or conjugate eye movement. Here we describe a digital camera-based eye tracking method “NeuroGaze” to capture the symmetry of eye movement while performing neurological eye examination. The proposed method was developed based on detecting the center of the pupil for both eyes from a given video and measuring eye conjugacy by transforming the pupil center coordinates to relative gaze. The method was tested on healthy volunteers while performing three neurological eye examinations <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">¹</xref> <fn id="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><label>¹</label> The NeuroEye dataset is made available at <uri>https://www.kaggle.com/datasets/mahassan8/neuroeye</uri> </fn> . We also compared our proposed approach to state-of-the-art digital camera-based eye-tracking methods and commercial off-the-shelf (COTS) eye trackers. NeuroGaze outperformed digital camera-based eye tracking methods by reporting a mean Spearman rank-order correlation coefficient of 0.86 for the H-test, 0.87 for the Dot-test, and 0.56 for the OKN-Test, and shows similarity in trends for the relative gaze trajectories with a noticeable offset in the scale of the relative gaze angle compared to COTS eye tracker (see Fig. 1). The study demonstrates that by using a pupil-center-based eye-tracking method, a digital camera can measure clinically relevant information regarding eye movement.

IS-CAT: Intensity-Spatial Cross-Attention Transformer for LiDAR-Based Place Recognition.

LiDAR place recognition is a crucial component of autonomous navigation, essential for loop closure in simultaneous localization and mapping (SLAM) systems. Notably, while camera-based methods struggle in fluctuating environments, such as weather or light, LiDAR demonstrates robustness against such challenges. This study introduces the intensity and spatial cross-attention transformer, which is a novel approach that utilizes LiDAR to generate global descriptors by fusing spatial and intensity data for enhanced place recognition. The proposed model leveraged a cross attention to a concatenation mechanism to process and integrate multi-layered LiDAR projections. Consequently, the previously unexplored synergy between spatial and intensity data was addressed. We demonstrated the performance of IS-CAT through extensive validation on the NCLT dataset. Additionally, we performed indoor evaluations on our Sejong indoor-5F dataset and demonstrated successful application to a 3D LiDAR SLAM system. Our findings highlight descriptors that demonstrate superior performance in various environments. This performance enhancement is evident in both indoor and outdoor settings, underscoring the practical effectiveness and advancements of our approach.

Fully Convolutional Neural Network for Vehicle Speed and Emergency-Brake Prediction.

Ego-vehicle state prediction represents a complex and challenging problem for self-driving and autonomous vehicles. Sensorial information and on-board cameras are used in perception-based solutions in order to understand the state of the vehicle and the surrounding traffic conditions. Monocular camera-based methods are becoming increasingly popular for driver assistance, with precise predictions of vehicle speed and emergency braking being important for road safety enhancement, especially in the prevention of speed-related accidents. In this research paper, we introduce the implementation of a convolutional neural network (CNN) model tailored for the prediction of vehicle velocity, braking events, and emergency braking, employing sequential image sequences and velocity data as inputs. The CNN model is trained on a dataset featuring sequences of 20 consecutive images and corresponding velocity values, all obtained from a moving vehicle navigating through road-traffic scenarios. The model's primary objective is to predict the current vehicle speed, braking actions, and the occurrence of an emergency-brake situation using the information encoded in the preceding 20 frames. We subject our proposed model to an evaluation on a dataset using regression and classification metrics, and comparative analysis with existing published work based on recurrent neural networks (RNNs). Through our efforts to improve the prediction accuracy for velocity, braking behavior, and emergency-brake events, we make a substantial contribution to improving road safety and offer valuable insights for the development of perception-based techniques in the field of autonomous vehicles.

Multi-Sensor Fusion Simultaneous Localization Mapping Based on Deep Reinforcement Learning and Multi-Model Adaptive Estimation.

In this study, we designed a multi-sensor fusion technique based on deep reinforcement learning (DRL) mechanisms and multi-model adaptive estimation (MMAE) for simultaneous localization and mapping (SLAM). The LiDAR-based point-to-line iterative closest point (PLICP) and RGB-D camera-based ORBSLAM2 methods were utilized to estimate the localization of mobile robots. The residual value anomaly detection was combined with the Proximal Policy Optimization (PPO)-based DRL model to accomplish the optimal adjustment of weights among different localization algorithms. Two kinds of indoor simulation environments were established by using the Gazebo simulator to validate the multi-model adaptive estimation localization performance, which is used in this paper. The experimental results of the proposed method in this study confirmed that it can effectively fuse the localization information from multiple sensors and enable mobile robots to obtain higher localization accuracy than the traditional PLICP and ORBSLAM2. It was also found that the proposed method increases the localization stability of mobile robots in complex environments.

Capturing the pulse: a state-of-the-art review on camera-based jugular vein assessment.

Heart failure is associated with a rehospitalisation rate of up to 50% within six months. Elevated central venous pressure may serve as an early warning sign. While invasive procedures are used to measure central venous pressure for guiding treatment in hospital, this becomes impractical upon discharge. A non-invasive estimation technique exists, where the clinician visually inspects the pulsation of the jugular veins in the neck, but it is less reliable due to human limitations. Video and signal processing technologies may offer a high-fidelity alternative. This state-of-the-art review analyses existing literature on camera-based methods for jugular vein assessment. We summarize key design considerations and suggest avenues for future research. Our review highlights the neck as a rich imaging target beyond the jugular veins, capturing comprehensive cardiac signals, and outlines factors affecting signal quality and measurement accuracy. Addressing an often quoted limitation in the field, we also propose minimum reporting standards for future studies.

Peripheral Oxygen Saturation Measurement Using an RGB Camera

In 2019, the COVID-19 pandemic became a serious issue around the world. Low blood oxygen is the most important symptom of COVID-19. However, many patients have no obvious respiratory symptoms and exhibit silent hypoxemia, which is typically not noticed by patients but can cause severe damage. Hypoxemia is also related to high-altitude illness, highlighting the importance of detecting hypoxemia for travelers in high-altitude areas. The most commonly used device for monitoring blood oxygen remains difficult to access for the general population. To address this problem, a few camera-based methods have been proposed. Nevertheless, these approaches are generally not robust and have not been evaluated completely. Therefore, we conduct a pilot study to show the feasibility of camera-based SpO <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> estimation, and then propose a contact-free system by which to measure SpO <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> using a general RGB camera. We adopt a k-nearest neighbor model as the backbone algorithm. For a comprehensive evaluation, we compiled two SpO <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> datasets: one collected at an altitude of 3,150 meters, and the other collected at 102 meters above sea level. Sixty subjects participated in the experiments, which included a mobile phone, a webcam, and an industrial camera. In a leave-one-subject-out validation, the proposed method respectively yields a mean absolute error of 4.39%, 4.45%, and 4.22% using the three cameras. In general, the proposed approach outperforms the benchmark algorithms. To our best knowledge, this is the first work to utilize real high-attitude data to address camera-based SpO <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> measurement.

Camera-based automated monitoring of flying insects in the wild (Camfi). II. flight behaviour and long-term population monitoring of migratory Bogong moths in Alpine Australia.

The Bogong moth Agrotis infusa is well known for its remarkable annual round-trip migration from its breeding grounds across eastern and southern Australia to its aestivation sites in the Australian Alps, to which it provides an important annual influx of nutrients. Over recent years, we have benefited from a growing understanding of the navigational abilities of the Bogong moth. Meanwhile, the population of Bogong moths has been shrinking. Recently, the ecologically and culturally important Bogong moth was listed as endangered by the IUCN Red List, and the establishment of a program for long-term monitoring of its population has been identified as critical for its conservation. Here, we present the results of two years of monitoring of the Bogong moth population in the Australian Alps using recently developed methods for automated wildlife-camera monitoring of flying insects, named Camfi. While in the Alps, some moths emerge from the caves in the evening to undertake seemingly random flights, filling the air with densities in the dozens per cubic metre. The purpose of these flights is unknown, but they may serve an important role in Bogong moth navigation. We found that these evening flights occur throughout summer and are modulated by daily weather factors. We present a simple heuristic model of the arrival to and departure from aestivation sites by Bogong moths, and confirm results obtained from fox-scat surveys which found that aestivating Bogong moths occupy higher elevations as the summer progresses. Moreover, by placing cameras along two elevational transects below the summit of Mt. Kosciuszko, we found that evening flights were not random, but were systematically oriented in directions relative to the azimuth of the summit of the mountain. Finally, we present the first recorded observations of the impact of bushfire smoke on aestivating Bogong moths - a dramatic reduction in the size of a cluster of aestivating Bogong moths during the fire, and evidence of a large departure from the fire-affected area the day after the fire. Our results highlight the challenges of monitoring Bogong moths in the wild and support the continued use of automated camera-based methods for that purpose.

Assessing Different Methods for Detection of Distracted Driving

Roadside observations provide insight into the prevalence of distracted driving on American roadways. Traditional observational studies require the use of observers stationed on the roadside who manually annotate the behavior of passing drivers, while camera-based methods allow researchers to code distracted driving from photos without an observer on the scene. The current study compared the accuracy of a traditional in-person roadside observer with a camera-based method in identifying driver distraction. Both methods distinguished cellphone distractions from other behaviors with high accuracy, although the in-person observer performed slightly better than the camera-based method across most specific behaviors. This study demonstrates that both in-person and camera-based observations are reasonably accurate approaches for measuring the prevalence of cellphone distractions on the road. Camera-based methods may provide an advantage to in-person observers by enabling observations on roadways where it is not feasible to station a human observer.

Real-time reading system for pointer meter based on YolactEdge

Despite the extensive deployment of digital instruments in modern times, their stability is challenging to maintain in adverse environmental conditions such as extreme temperatures, pressure, or powerful electromagnetic radiation. Analog meters, owing to their mechanical resilience and electromagnetic impedance, persistently find usage across nuclear power plants, petroleum, and chemical industries. However, under these harsh conditions, manual reading of the instruments may prove to be difficult and dangerous while failing to meet the requirements of real-time monitoring. In recent years, several machine vision-based meter reading systems have been proposed, however, achieving high accuracy through camera-based methods under varying angles and lighting conditions poses significant challenges. Cloud deployment may compromise plant privacy, while edge computing faces limitations in real-time meter reading due to limited computing power. To address these issues, we propose a real-time reading system based on the YolactEdge instance segmentation framework for single-point analog meters. Our system is more accurate than previous studies and is implemented and deployed on the Jetson Xavier NX edge computing device. Our performance evaluation shows that our model outperforms other baselines, with low reference values and relative errors of 0.0237% and 0.0300%, respectively, and an average inference speed of 10.26 FPS with INIT 8 linear acceleration on Nvidia Jetson NX.

Brass Haptics: Comparing Virtual and Physical Trumpets in Extended Realities

Despite the benefits of learning an instrument, many students drop out early because it can be frustrating for the student, expensive for the caregiver, and loud for the household. Virtual Reality (VR) and Extended Reality (XR) offer the potential to address these challenges by simulating multiple instruments in an engaging and motivating environment through headphones. To assess the potential for commercial VR to augment musical experiences, we used standard VR implementation processes to design four virtual trumpet interfaces: camera-tracking with tracked register selection (two ways), camera-tracking with voice activation, and a controller plus a force-feedback haptic glove. To evaluate these implementations, we created a virtual music classroom that produces audio, notes, and finger pattern guides loaded from a selected Musical Instrument Digital Interface (MIDI) file. We analytically compared these implementations against physical trumpets (both acoustic and MIDI), considering features of ease of use, familiarity, playability, noise, and versatility. The physical trumpets produced the most reliable and familiar experience, and some XR benefits were considered. The camera-based methods were easy to use but lacked tactile feedback. The haptic glove provided improved tracking accuracy and haptic feedback over camera-based methods. Each method was also considered as a proof-of-concept for other instruments, real or imaginary.

Resolution-independent liveness detection via computational ghost imaging

The security of liveness detection in face recognition is a crucial issue, but many attacks can spoof current face feature techniques. To enhance the security of liveness detection, a method is proposed to extract human physiological components from the object and classify the properties. The proposed method, different from traditional camera-based methods that require specific movement of the human face, separates the heart rate (HR) components from the computational ghost imaging (CGI) signal and achieves liveness detection by capturing only one image rather than image sequences. The correct rate reaches 96.0% against picture attacks and mask attacks. The average error is only 3.57% compared to commercial contact HR measuring devices. Meanwhile, this method is found resolution-independent and can work in low-resolution conditions, which is experimentally verified at a resolution of 32 × 32 pixels. This method can enhance the security of liveness detection and provide a fresh framework for physiological measurements.

Towards the automated virtual trial assembly of large and complex steel members using terrestrial laser scanning and BIM

Virtual Trial Assembly (VTA) is proposed as an alternative to physical trial assembly that can significantly reduce cost and time requirements. Existing studies on VTA can be categorized as total station and camera-based methods or Terrestrial Laser Scanning (TLS)-based methods. The real coordinate measurement procedure in existing total station and camera-based VTA methods is inefficient and complex because many targets have to be manually installed at assembly points and because different members require different installation schemes. Existing TLS-based VTA methods involve significant human intervention because assembly point extraction and assembly simulation are achieved with commercial software. In addition, geometric quality inspection is not performed in TLS-based VTA methods, although the basis for VTA is the dimensional quality compliance of the components. Therefore, this study presents an automated approach to perform VTA for large and complex steel members based on TLS, and Building Information Modeling (BIM) is utilized to provide as-designed geometric data. Specifically, an automated method is presented to extract assembly points. An internal point removal method is proposed for a BIM model to improve the accuracy of fine registration. A geometric quality inspection method is developed for bending and torsional deviation inspection. The proposed approach is found to be reliable through its application to a practical super high-rise building.

Using Video Cameras as a Research Tool in Public Spaces: Addressing Ethical and Information Governance Challenges Under Data Protection Legislation

Systematic observation is a promising unobtrusive method of assessing human behavior in urban environments without many issues typically associated with self-report measures (e.g., recall bias, low response rates). Improvements in video camera technologies make it more feasible for researchers to conduct systematic observation, which could reduce the time, labor, and cost to facilitate high-quality observational research in urban environments at scale. However, there are important ethical and information governance challenges driven by data protection laws, which discourage many researchers from using camera-based observation methods. The European Union General Data Protection Regulation is a leading global standard for data protection. Drawing on our experiences of conducting three studies using video cameras in public spaces, we discuss how to conduct this kind of research in line with General Data Protection Regulation requirements. The paper outlines issues concerning data protection, privacy, informed consent, and confidentiality, and how we addressed them. In doing this, the paper provides support for responsible use of camera-based observation methods, which will be of value to researchers, ethics committees, and funders. Outlining how to use video cameras responsibly will enable more research to be conducted that, in turn, will build the case for its benefits to researchers and society.