Depth Video Research Articles

Depth Video-Based Secondary Action Recognition in Vehicles via Convolutional Neural Network and Bidirectional Long Short-Term Memory with Spatial Enhanced Attention Mechanism

Secondary actions in vehicles are activities that drivers engage in while driving that are not directly related to the primary task of operating the vehicle. Secondary Action Recognition (SAR) in drivers is vital for enhancing road safety and minimizing accidents related to distracted driving. It also plays an important part in modern car driving systems such as Advanced Driving Assistance Systems (ADASs), as it helps identify distractions and predict the driver’s intent. Traditional methods of action recognition in vehicles mostly rely on RGB videos, which can be significantly impacted by external conditions such as low light levels. In this research, we introduce a novel method for SAR. Our approach utilizes depth-video data obtained from a depth sensor located in a vehicle. Our methodology leverages the Convolutional Neural Network (CNN), which is enhanced by the Spatial Enhanced Attention Mechanism (SEAM) and combined with Bidirectional Long Short-Term Memory (Bi-LSTM) networks. This method significantly enhances action recognition ability in depth videos by improving both the spatial and temporal aspects. We conduct experiments using K-fold cross validation, and the experimental results show that on the public benchmark dataset Drive&Act, our proposed method shows significant improvement in SAR compared to the state-of-the-art methods, reaching an accuracy of about 84% in SAR in depth videos.

TRRHA: A two-stream re-parameterized refocusing hybrid attention network for synthesized view quality enhancement

In multi-view video systems, the decoded texture video and its corresponding depth video are utilized to synthesize virtual views from different perspectives using the depth-image-based rendering (DIBR) technology in 3D-high efficiency video coding (3D-HEVC). However, the distortion of the compressed multi-view video and the disocclusion problem in DIBR can easily cause obvious holes and cracks in the synthesized views, degrading the visual quality of the synthesized views. To address this problem, a novel two-stream re-parameterized refocusing hybrid attention (TRRHA) network is proposed to significantly improve the quality of synthesized views. Firstly, a global multi-scale residual information stream is applied to extract the global context information by using refocusing attention module (RAM), and the RAM can detect the contextual feature and adaptively learn channel and spatial attention feature to selectively focus on different areas. Secondly, a local feature pyramid attention information stream is used to fully capture complex local texture details by using re-parameterized refocusing attention module (RRAM). The RRAM can effectively capture multi-scale texture details with different receptive fields, and adaptively adjust channel and spatial weights to adapt to information transformation at different sizes and levels. Finally, an efficient feature fusion module is proposed to effectively fuse the extracted global and local information streams. Extensive experimental results show that the proposed TRRHA achieves significantly better performance than the state-of-the-art methods. The source code will be available at https://github.com/647-bei/TRRHA.

465 Comparative analysis of semantic segmentation and deep regression models with supervised pre-training for accurate prediction of pig body weight from video data: Insights from industry-scale datasets

Abstract Accurate pig body weight (BW) measurement is essential for producers as it is related to pig growth, health, and marketing, yet conventional manual weighing methods are time-consuming and may cause potential stress to the animals. Although there is a growing trend towards the adoption of three-dimensional cameras coupled with computer vision techniques for pig BW estimation, their validation using industry-scale data are still limited. Among the prevailing methodologies, semantic segmentation and supervised pre-training regression are prominent models currently used. Therefore, the objectives of this study were: 1) to estimate pig BW from repeatedly measured video data obtained from a commercial setting, 2) to compare the performance of the two image analysis methods: thresholding segmentation and deep regression approaches, and 3) to evaluate the predictive ability of BW estimation models. An Intel RealSense D435 camera was installed in the commercial farm to collect top-view videos of 540 pigs biweekly at five different time points over three months. At the same time, manually measured BW records were collected using a digital weighing system. We used an automated video conversion pipeline and fine-tuned YOLOv8 to pre-process the raw depth videos. Subsequently, we acquired a total of 151,756 depth images and depth map files. Adaptive thresholding was applied to segment the pig body from the background. Four image-derived biometric features, including dorsal length, abdominal width, height, and volume, were estimated from the segmented images and fitted using ordinary least squares and random forest models. We applied transfer learning by initializing the weights of five deep learning models, including ResNet50, Xception, EfficientNetV2S, ConvNeXtBase, and Vision Transformer, with pre-trained weights from ImageNet. We then fine-tuned these models on the pig depth images. The last layer of each model was adapted to linear regression, enabling direct estimation of BW for each image without the need for additional image pre-processing steps. We employed random repeated subsampling cross-validation, dividing 80% of the pigs for training and 20% for testing, to evaluate prediction performance at each time point. The best prediction coefficients of determination and mean absolute percentage error for each time point were 0.76, 0.86, 0.90, 0.83, 0.90, and 4.57%, 3.76%, 3.01%, 3.41%, 4.84%, respectively. On average, the Xception model resulted in the best prediction coefficient of determination and mean absolute percentage error of 0.90 and 3.01%. Our results suggest that deep learning-based supervised learning models improve the prediction performance of pig BW from industry-scale depth video data.

Lower limb joint contact forces and ground reaction forces analysis based on Azure Kinect motion capture

Traditional gait analysis systems are typically complex to operate, lack portability, and involve high equipment costs. This study aims to establish a musculoskeletal dynamics calculation process driven by Azure Kinect. Building upon the full-body model of the Anybody musculoskeletal simulation software and incorporating a foot-ground contact model, the study utilized Azure Kinect-driven skeletal data from depth videos of 10 participants. The in-depth videos were prepossessed to extract keypoint of the participants, which were then adopted as inputs for the musculoskeletal model to compute lower limb joint angles, joint contact forces, and ground reaction forces. To validate the Azure Kinect computational model, the calculated results were compared with kinematic and kinetic data obtained using the traditional Vicon system. The forces in the lower limb joints and the ground reaction forces were normalized by dividing them by the body weight. The lower limb joint angle curves showed a strong correlation with Vicon results (mean ρ values: 0.78 ~ 0.92) but with root mean square errors as high as 5.66°. For lower limb joint force prediction, the model exhibited root mean square errors ranging from 0.44 to 0.68, while ground reaction force root mean square errors ranged from 0.01 to 0.09. The established musculoskeletal dynamics model based on Azure Kinect shows good prediction capabilities for lower limb joint forces and vertical ground reaction forces, but some errors remain in predicting lower limb joint angles.

A validation study demonstrating portable motion capture cameras accurately characterize gait metrics when compared to a pressure-sensitive walkway

Digital quantification of gait can be used to measure aging- and disease-related decline in mobility. Gait performance also predicts prognosis, disease progression, and response to therapies. Most gait analysis systems require large amounts of space, resources, and expertise to implement and are not widely accessible. Thus, there is a need for a portable system that accurately characterizes gait. Here, depth video from two portable cameras accurately reconstructed gait metrics comparable to those reported by a pressure-sensitive walkway. 392 research participants walked across a four-meter pressure-sensitive walkway while depth video was recorded. Gait speed, cadence, and step and stride durations and lengths strongly correlated (r > 0.9) between modalities, with root-mean-squared-errors (RMSE) of 0.04 m/s, 2.3 steps/min, 0.03 s, and 0.05–0.08 m for speed, cadence, step/stride duration, and step/stride length, respectively. Step, stance, and double support durations (gait cycle percentage) significantly correlated (r > 0.6) between modalities, with 5% RMSE for step and stance and 10% RMSE for double support. In an exploratory analysis, gait speed from both modalities significantly related to healthy, mild, moderate, or severe categorizations of Charleson Comorbidity Indices (ANOVA, Tukey’s HSD, p < 0.0125). These findings demonstrate the viability of using depth video to expand access to quantitative gait assessments.

Deep degradation-aware up-sampling-based depth video coding

Deep degradation-aware up-sampling-based depth video coding

Selfredepth

Depth maps produced by consumer-grade sensors suffer from inaccurate measurements and missing data from either system or scene-specific sources. Data-driven denoising algorithms can mitigate such problems; however, they require vast amounts of ground truth depth data. Recent research has tackled this limitation using self-supervised learning techniques, but it requires multiple RGB-D sensors. Moreover, most existing approaches focus on denoising single isolated depth maps or specific subjects of interest highlighting a need for methods that can effectively denoise depth maps in real-time dynamic environments. This paper extends state-of-the-art approaches for depth-denoising commodity depth devices, proposing SelfReDepth, a self-supervised deep learning technique for depth restoration, via denoising and hole-filling by inpainting of full-depth maps captured with RGB-D sensors. The algorithm targets depth data in video streams, utilizing multiple sequential depth frames coupled with color data to achieve high-quality depth videos with temporal coherence. Finally, SelfReDepth is designed to be compatible with various RGB-D sensors and usable in real-time scenarios as a pre-processing step before applying other depth-dependent algorithms. Our results demonstrate our approach’s real-time performance on real-world datasets shows that it outperforms state-of-the-art methods in denoising and restoration performance at over 30 fps on Commercial Depth Cameras, with potential benefits for augmented and mixed-reality applications.

Characterizing the structure of mouse behavior using Motion Sequencing.

Spontaneous mouse behavior is composed from repeatedly used modules of movement (e.g., rearing, running or grooming) that are flexibly placed into sequences whose content evolves over time. By identifying behavioral modules and the order in which they are expressed, researchers can gain insight into the effect of drugs, genes, context, sensory stimuli and neural activity on natural behavior. Here we present a protocol for performing Motion Sequencing (MoSeq), an ethologically inspired method that uses three-dimensional machine vision and unsupervised machine learning to decompose spontaneous mouse behavior into a series of elemental modules called 'syllables'. This protocol is based upon a MoSeq pipeline that includes modules for depth video acquisition, data preprocessing and modeling, as well as a standardized set of visualization tools. Users are provided with instructions and code for building a MoSeq imaging rig and acquiring three-dimensional video of spontaneous mouse behavior for submission to the modeling framework; the outputs of this protocol include syllable labels for each frame of the video data as well as summary plots describing how often each syllable was used and how syllables transitioned from one to the other. In addition, we provide instructions for analyzing and visualizing the outputs of keypoint-MoSeq, a recently developed variant of MoSeq that can identify behavioral motifs from keypoints identified from standard (rather than depth) video. This protocol and the accompanying pipeline significantly lower the bar for users without extensive computational ethology experience to adopt this unsupervised, data-driven approach to characterize mouse behavior.

Depth over RGB: automatic evaluation of open surgery skills using depth camera

PurposeIn this paper, we present a novel approach to the automatic evaluation of open surgery skills using depth cameras. This work is intended to show that depth cameras achieve similar results to RGB cameras, which is the common method in the automatic evaluation of open surgery skills. Moreover, depth cameras offer advantages such as robustness to lighting variations, camera positioning, simplified data compression, and enhanced privacy, making them a promising alternative to RGB cameras.MethodsExperts and novice surgeons completed two simulators of open suturing. We focused on hand and tool detection and action segmentation in suturing procedures. YOLOv8 was used for tool detection in RGB and depth videos. Furthermore, UVAST and MSTCN++ were used for action segmentation. Our study includes the collection and annotation of a dataset recorded with Azure Kinect.ResultsWe demonstrated that using depth cameras in object detection and action segmentation achieves comparable results to RGB cameras. Furthermore, we analyzed 3D hand path length, revealing significant differences between experts and novice surgeons, emphasizing the potential of depth cameras in capturing surgical skills. We also investigated the influence of camera angles on measurement accuracy, highlighting the advantages of 3D cameras in providing a more accurate representation of hand movements.ConclusionOur research contributes to advancing the field of surgical skill assessment by leveraging depth cameras for more reliable and privacy evaluations. The findings suggest that depth cameras can be valuable in assessing surgical skills and provide a foundation for future research in this area.

Corr-Track: Category-Level 6D Pose Tracking with Soft-Correspondence Matrix Estimation.

Category-level pose tracking methods can continuously track the pose of objects without requiring any prior knowledge of the specific shape of the tracked instance. This makes them advantageous in augmented reality and virtual reality applications. The key challenge is how to train neural networks to accurately predict the poses of objects they have never seen before and exhibit strong generalization performance. We propose a novel category-level 6D pose tracking method Corr-Track, which is capable of accurately tracking objects belonging to the same category from depth video streams. Our approach utilizes direct soft correspondence constraints to train a neural network, which estimates bidirectional soft correspondences between sparsely sampled point clouds of objects in two frames. We first introduce a soft correspondence matrix for pose tracking tasks and establish effective constraints through direct spatial point-to-point correspondence representations in the sparse point cloud correspondence matrix. We propose the "point cloud expansion" strategy to address the "point cloud shrinkage" problem resulting from soft correspondences. This strategy ensures that the corresponding point cloud accurately reproduces the shape of the target point cloud, leading to precise pose tracking results. We evaluated our approach on the NOCS-REAL275 and Wild6D dataset and observed superior performance compared to previous methods. Additionally, we conducted cross-category experiments that further demonstrated its generalization capability.

Enhancing automated lower limb rehabilitation exercise task recognition through multi-sensor data fusion in tele-rehabilitation

BackgroundTele-rehabilitation is the provision of physiotherapy services to individuals in their own homes. Activity recognition plays a crucial role in the realm of automatic tele-rehabilitation. By assessing patient movements, identifying exercises, and providing feedback, these platforms can offer insightful information to clinicians, thereby facilitating an improved plan of care. This study introduces a novel deep learning approach aimed at identifying lower limb rehabilitation exercises. This is achieved through the integration of depth data and pressure heatmaps. We hypothesized that combining pressure heatmaps and depth data could improve the model’s overall performance.MethodsIn this study, depth videos and body pressure data from an accessible online dataset were used. This dataset comprises data from 30 healthy individuals performing 7 lower limb rehabilitation exercises. To accomplish the classification task, three deep learning models were developed, all based on an established 3D-CNN architecture. The models were designed to classify the depth videos, sequences of pressure data frames, and combination of depth videos and pressure frames. The models’ performance was assessed through leave-one-subject-out and leave-multiple-subjects-out cross-validation methods. Performance metrics, including accuracy, precision, recall, and F1 score, were reported for each model.ResultsOur findings indicated that the model trained on the fusion of depth and pressure data showed the highest and most stable performance when compared with models using individual modality inputs. This model could effectively identify the exercises with an accuracy of 95.71%, precision of 95.83%, recall of 95.71%, and an F1 score of 95.74%.ConclusionOur results highlight the impact of data fusion for accurately classifying lower limb rehabilitation exercises. We showed that our model could capture different aspects of exercise movements using the visual and weight distribution data from the depth camera and pressure mat, respectively. This integration of data provides a better representation of exercise patterns, leading to higher classification performance. Notably, our results indicate the potential application of this model in automatic tele-rehabilitation platforms.

Revolutionizing Cow Welfare Monitoring: A Novel Top-View Perspective with Depth Camera-Based Lameness Classification.

This study innovates livestock health management, utilizing a top-view depth camera for accurate cow lameness detection, classification, and precise segmentation through integration with a 3D depth camera and deep learning, distinguishing it from 2D systems. It underscores the importance of early lameness detection in cattle and focuses on extracting depth data from the cow's body, with a specific emphasis on the back region's maximum value. Precise cow detection and tracking are achieved through the Detectron2 framework and Intersection Over Union (IOU) techniques. Across a three-day testing period, with observations conducted twice daily with varying cow populations (ranging from 56 to 64 cows per day), the study consistently achieves an impressive average detection accuracy of 99.94%. Tracking accuracy remains at 99.92% over the same observation period. Subsequently, the research extracts the cow's depth region using binary mask images derived from detection results and original depth images. Feature extraction generates a feature vector based on maximum height measurements from the cow's backbone area. This feature vector is utilized for classification, evaluating three classifiers: Random Forest (RF), K-Nearest Neighbor (KNN), and Decision Tree (DT). The study highlights the potential of top-view depth video cameras for accurate cow lameness detection and classification, with significant implications for livestock health management.

Insights on Using Time-of-Flight Camera for Recovering Cardiac Pulse From Chest Motion in Depth Videos.

In this article, we introduce a novel use of depth camera to extract cardiac pulse signal from human chest area, in which the depth information is obtained from a near infrared sensor using time-of-flight technology. We successfully isolate weak chest motion due to heartbeat by processing a sequence of depth images without raising privacy concern. We discuss motion sensitivity in depth video with examples from actuator simulation and human chest motion. Compared to other imaging modalities, the depth image intensity can be directly used for micromotion reconstruction. To deal with the challenges of recovering heartbeat from the chest area, we develop a set of coherent processing techniques to suppress the unwanted motion interference from breathing motion and involuntary body motion and eventually obtain clean cardiac pulse signal. We, thus, derive inter-beat-interval, showing high consistency to the contact photoplethysmography. Additionally, we develop a graphical interpretation of the most and the less pulsatile principal components in eigen space. For validation, we test our method on ten healthy human subjects with different resting heart rates. More importantly, we conduct a set of experiments to study the robustness and weakness of our methods, including extended range, multi-subject, thickness of clothes and generation to other measurement site.

CUSCO: An Unobtrusive Custom Secure Audio-Visual Recording System for Ambient Assisted Living.

The ubiquity of digital technology has facilitated detailed recording of human behaviour. Ambient technology has been used to capture behaviours in a broad range of applications ranging from healthcare and monitoring to assessment of cooperative work. However, existing systems often face challenges in terms of autonomy, usability, and privacy. This paper presents a portable, easy-to-use and privacy-preserving system for capturing behavioural signals unobtrusively in home or in office settings. The system focuses on the capture of audio, video, and depth imaging. It is based on a device built on a small-factor platform that incorporates ambient sensors which can be integrated with the audio and depth video hardware for multimodal behaviour tracking. The system can be accessed remotely and integrated into a network of sensors. Data are encrypted in real time to ensure safety and privacy. We illustrate uses of the device in two different settings, namely, a healthy-ageing IoT application, where the device is used in conjunction with a range of IoT sensors to monitor an older person's mental well-being at home, and a healthcare communication quality assessment application, where the device is used to capture a patient-clinician interaction for consultation quality appraisal. CUSCO can automatically detect active speakers, extract acoustic features, record video and depth streams, and recognise emotions and cognitive impairment with promising accuracy.

MouseVUER: video based open-source system for laboratory mouse home-cage monitoring

Video monitoring of mice in the home-cage reveals behavior profiles without the disruptions caused by specialized test setups and makes it possible to quantify changes in behavior patterns continually over long time frames. Several commercial home-cage monitoring systems are available with varying costs and capabilities; however there are currently no open-source systems for home-cage monitoring. We present an open-source system for top-down video monitoring of research mice in a slightly modified home-cage. The system is designed for integration with Allentown NexGen ventilated racks and allows unobstructed view of up to three mice, but can also be operated outside the rack. The system has an easy to duplicate and assemble home-cage design along with a video acquisition solution. The system utilizes a depth video camera, and we demonstrate the robustness of depth video for home-cage mice monitoring. For researchers without access to Allentown NexGen ventilated racks, we provide designs and assembly instructions for a standalone non-ventilated rack solution that holds three systems for more compact and efficient housing. We make all the design files, along with detailed assembly and installation instructions, available on the project webpage (https://github.com/NIH-CIT-OIR-SPIS/MouseVUER).

Distributed Computation for Sparse Recovery via Continuous-Time Neurodynamic Approach

Sparsity has been extensively employed in multimedia sensing and computing in consumer electronics, signal and image processing, depth video codec, adaptive sparse-type equalizer, blind speech separation, and machine learning. Throughout this paper, we propose a novel distributed projection neurodynamic approach for solving the Basis Pursuit (BP) with flexible partition methods in a distributed manner. The proposed neurodynamic approach requires only that the network is undirected and connected, and no node can access the entire matrix simultaneously. First, we equivalently formulate the BP into a standard distributed optimization problem with a flexible partition-by-blocks method to obtain global information, and discuss the equivalence of their optimality conditions. Then, we propose a distributed continuous-time neurodynamic approach on the basis of primal-dual dynamical systems and projection operators, and also study its global convergence property. Finally, numerical experiments on sparse signals and image recovery further verify the effectiveness and superiority of our proposed neurodynamic approach.

Joint angle estimation during shoulder abduction exercise using contactless technology

BackgroundTele-rehabilitation, also known as tele-rehab, uses communication technologies to provide rehabilitation services from a distance. The COVID-19 pandemic has highlighted the importance of tele-rehab, where the in-person visits declined and the demand for remote healthcare rises. Tele-rehab offers enhanced accessibility, convenience, cost-effectiveness, flexibility, care quality, continuity, and communication. However, the current systems are often not able to perform a comprehensive movement analysis. To address this, we propose and validate a novel approach using depth technology and skeleton tracking algorithms.MethodsOur data involved 14 participants (8 females, 6 males) performing shoulder abduction exercises. We collected depth videos from an LiDAR camera and motion data from a Motion Capture (Mocap) system as our ground truth. The data were collected at distances of 2 m, 2.5 m, and 3.5 m from the LiDAR sensor for both arms. Our innovative approach integrates LiDAR with the Cubemos and Mediapipe skeleton tracking frameworks, enabling the assessment of 3D joint angles. We validated the system by comparing the estimated joint angles versus Mocap outputs. Personalized calibration was applied using various regression models to enhance the accuracy of the joint angle calculations.ResultsThe Cubemos skeleton tracking system outperformed Mediapipe in joint angle estimation with higher accuracy and fewer errors. The proposed system showed a strong correlation with Mocap results, although some deviations were present due to noise. Precision decreased as the distance from the camera increased. Calibration significantly improved performance. Linear regression models consistently outperformed nonlinear models, especially at shorter distances.ConclusionThis study showcases the potential of a marker-less system, to proficiently track body joints and upper-limb angles. Signals from the proposed system and the Mocap system exhibited robust correlation, with Mean Absolute Errors (MAEs) consistently below 10∘\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$10^\\circ$$\\end{document}. LiDAR’s depth feature enabled accurate computation of in-depth angles beyond the reach of traditional RGB cameras. Altogether, this emphasizes the depth-based system’s potential for precise joint tracking and angle calculation in tele-rehab applications.

Recognition and detection of unusual activities in ATM using dual-channel capsule generative adversarial network

Real-time detection of human activity has become essential for monitoring and security of public spaces such as bank ATMs and workplaces, due to daily rise in criminal activities. Currently, monocular CCTV cameras that only record RGB video are used for monitoring such restricted areas. In addition to RGB data, the RGB + D sensor also offers depth information about the scene. To overcome the issue of online recognition of anomalous activities in Bank ATMs, a supervised deep learning method utilizing Dual-Channel Capsule Generative Adversarial Network (DCCGAN) and RGB + D sensor is proposed. The input RGB + D Dataset is given to super pixel motion detection method to find the region of interest (ROI). Motion detection is a significant stage examination of wide scene for background subtraction and foreground detection. After arranging the motion detection, its region is positioned frame by frame. Then, the detected ROI is given to fast discrete curvelet transform with the wrapping (FDCT-WRP) method. FDCT-WRP feature extraction method. These extracting features are supplied to Deep Convolutional Spiking neural network (DCSNN), which realize the object. RGB and depth video segments are used to create motion templates from the RGB + D data online video stream. These templates are trained on DCCGANs to identify distrustful events in current activity and categorized as normal and abnormal. Additionally, a unique RGB + D dataset is employed because there was no existing dataset available for analyzing human activity in ATMs. The proposed DUA-DCCGAN-ATM approach is assessed on qualitative with quantitative statistical evaluation parameters and identify suspect occurrence with 0.932 precision and 98.2 % accuracy. The detailed statistical analysis exemplifies that the proposed technique can identify distrustful events in a real-time online manner prior completing the anomalous activity.

Spatio-Temporal Information Fusion and Filtration for Human Action Recognition

Human action recognition (HAR) as the most representative human-centred computer vision task is critical in human resource management (HRM), especially in human resource recruitment, performance appraisal, and employee training. Currently, prevailing approaches to human action recognition primarily emphasize either temporal or spatial features while overlooking the intricate interplay between these two dimensions. This oversight leads to less precise and robust action classification within complex human resource recruitment environments. In this paper, we propose a novel human action recognition methodology for human resource recruitment environments, which aims at symmetrically harnessing temporal and spatial information to enhance the performance of human action recognition. Specifically, we compute Depth Motion Maps (DMM) and Depth Temporal Maps (DTM) from depth video sequences as space and time descriptors, respectively. Subsequently, a novel feature fusion technique named Center Boundary Collaborative Canonical Correlation Analysis (CBCCCA) is designed to enhance the fusion of space and time features by collaboratively learning the center and boundary information of feature class space. We then introduce a spatio-temporal information filtration module to remove redundant information introduced by spatio-temporal fusion and retain discriminative details. Finally, a Support Vector Machine (SVM) is employed for human action recognition. Extensive experiments demonstrate that the proposed method has the ability to significantly improve human action recognition performance.

A Couch Mounted Smartphone-based Motion Monitoring System for Radiation Therapy

PurposeSurface-guided radiation-therapy (SGRT) systems are being adopted into clinical practice for patient setup and motion monitoring. However, commercial systems remain cost prohibitive to resource-limited clinics around the world. Our aim is to develop and validate a smartphone-based application using LiDAR cameras (such as on recent Apple iOS devices) for facilitating SGRT in low-resource centers. The proposed SGRT application was tested at multiple institutions, and validated using phantoms and volunteers against various commercial systems to demonstrate feasibility. Methods and MaterialsAn iOS application was developed in Xcode and written in Swift using the Augmented-Reality (AR) Kit and implemented on an Apple iPhone 13 Pro with a built-in LiDAR camera. The application contains multiple features: 1) visualization of both the camera and depth video feeds (at a ∼60Hz sample-frequency), 2) region-of-interest (ROI) selection over the patient's anatomy where motion is measured, 3) chart displaying the average motion over time in the ROI, and 4) saving/exporting the motion traces and surface map over the ROI for further analysis. The iOS application was tested to evaluate depth measurement accuracy for: 1) different angled surfaces, 2) different field-of-views over different distances, and 3) similarity to a commercially available SGRT systems (Vision RT AlignRT® and Varian IDENTIFY™) with motion phantoms and healthy volunteers across three institutions. Measurements were analyzed using linear-regressions and Bland-Altman analysis. ResultsCompared to the clinical system measurements (reference), the iOS application showed excellent agreement for depth (r=1.000,p<0.0001; bias=-0.07±0.24cm) and angle (r=1.000,p<0.0001; bias=0.02±0.69°) measurements. For free-breathing traces, the iOS application was significantly correlated to phantom motion (institute 1: r=0.99,p<0.0001; bias=-0.003±0.03cm; institute 2: r=0.98,p<0.0001; bias=-0.001±0.10cm; institute 3: r=0.97,p<0.0001; bias=0.04±0.06cm) and healthy volunteer motion (institute 1: r=0.98,p<0.0001; bias=-0.008±0.06cm; institute 2: r=0.99,p<0.0001; bias=-0.007±0.12cm; institute 3: r=0.99,p<0.0001; bias=-0.001±0.04cm). ConclusionThe proposed approach using a smartphone-based application provides a low-cost platform that could improve access to surface-guided radiation therapy accounting for motion.