3D Video Research Articles

Deep-Learning-Based Approaches for Semantic Segmentation of Natural Scene Images: A Review

The task of semantic segmentation holds a fundamental position in the field of computer vision. Assigning a semantic label to each pixel in an image is a challenging task. In recent times, significant advancements have been achieved in the field of semantic segmentation through the application of Convolutional Neural Networks (CNN) techniques based on deep learning. This paper presents a comprehensive and structured analysis of approximately 150 methods of semantic segmentation based on CNN within the last decade. Moreover, it examines 15 well-known datasets in the semantic segmentation field. These datasets consist of 2D and 3D image and video frames, including general, indoor, outdoor, and street scenes. Furthermore, this paper mentions several recent techniques, such as SAM, UDA, and common post-processing algorithms, such as CRF and MRF. Additionally, this paper analyzes the performance evaluation of reviewed state-of-the-art methods, pioneering methods, common backbone networks, and popular datasets. These have been compared according to the results of Mean Intersection over Union (MIoU), the most popular evaluation metric of semantic segmentation. Finally, it discusses the main challenges and possible solutions and underlines some future research directions in the semantic segmentation task. We hope that our survey article will be useful to provide a foreknowledge to the readers who will work in this field.

External validation of a deep learning algorithm for automated echocardiographic strain measurements

Abstract Funding Acknowledgements Type of funding sources: Private company. Main funding source(s): Us2ai. Background Echocardiographic strain imaging reflects myocardial deformation and is a sensitive measure of cardiac function and wall-motion abnormalities. Manual strain analysis is time-consuming, prone to error and requires considerable expertise. Deep learning (DL) algorithms could automate interpretation of echocardiographic strain imaging. Purpose To develop and externally validate automated DL-based measurement of the left ventricle (LV) global longitudinal strain (GLS) and regional strain. Methods We developed and validated an automated DL workflow using convolutional neural networks to automatically annotate and assess individual 2D videos of apical 4-chamber, 2-chamber and 3-chamber views. We trained the algorithm in an internal dataset and validated GLS externally in (1) a real-world general population dataset from Taiwan, (2) a core-lab measured dataset from the multinational PROMIS-HFpEF study, and regional strain in (3) the HMC-QU-MI study of patients with suspected myocardial infarction (MI). Results In the external datasets, the DL workflow successfully analysed 3741 (89%) studies in the real-world cohort, 176 (96%) in PROMIS-HFpEF and 158 (98%) in HMC-QU-MI. Automated GLS showed good agreement with manual measurements in the real-world dataset (mean ± SD −18.9±4.5% vs −.18.2±4.4%, respectively; bias 0.7±2.5%, root-mean-squared error [RMSE] 2.6, Pearson’s correlation 0.84; Figure 1A) and in PROMIS-HFpEF (−15.4±4.1% vs −15.9±3.6%, respectively; bias −0.6±2.7%, RMSE 2.8, Pearson’s correlation 0.76; Figure 1B). In the general population cohort, automated GLS measurements accurately identified patients with vs without heart failure (ROC-AUC 0.89 for total HF and 0.98 for HF with reduced ejection fraction). In patients with suspected MI, automated regional strain identified regional wall-motion abnormalities with average ROC-AUC 0.80 (range 0.74–0.86 for apical 4-chamber view [Figure 2A] and 0.69–0.90 for apical 2-chamber view [Figure 2B]). Conclusions DL algorithms can interpret cardiac strain images with similar accuracy compared with conventional measurements. These results highlight the potential of DL algorithms to democratize the use of cardiac strain measurements.

An Indispensable Revolution For Pakistan, The Beginning Of Robot Assisted MIMVS.

Madam, Minimally invasive surgical (MIS) techniques or ‘sternal sparing’ strategies, performed by making minor right chest incisions, were introduced in the mid-1990s [1,3]. From that point forward, there has been a decrease in surgical injury and postoperative recuperation, coming about in expanded acknowledgement of these approaches. It works on three essential standards: rebuilding and conserving MV pamphlet portability, creating a vast flyer coaptation surface, and remodeling of the mitral annulus to supply an ideal and steady orifice area. [1] The history of MIMVS expanded from the 1990s when minimally invasive approaches such as the parasternal incision, hemi sternal incision and the mini thoracotomy were investigated by autonomous groups led by Delos Cosgrove and Lawrence Cohn. Further advancements include the primary video-directed repair and substitutions performed through a mini-thoracotomy and the primary utilization of end balloon clamping. Later, improvements incorporated the coming of stereoscopic 3D video endoscopy and mechanical surgery. [5] Robotic technology facilitates complex surgical procedures on the mitral valve without opening the chest. The most significant advantage is diminished surgical injury, which deciphers into early recuperation. In 1998, robotic mitral valve repair was initiated and adopted in 2005 by the US following the first multicenter trial. In classic robotic mitral valve surgery, femoral-femoral cardiopulmonary machine cannulation is performed through a small incision in the groin. The surgeon leading the robotic mitral valve surgery must have the relevant heart surgery certification from the country's certifying institution. The console surgeon should be experienced in all aspects of mitral valve surgery and be highly versed in repair techniques as robotic procedures take longer than traditional surgeries, especially during the learning curve. Apprenticeship in the so-called mini-mitral using conventional videoscopy is also an advantage, but not a mandatory requirement The courses on robotic mitral valve surgery including formal didactic and practice-oriented training should be obtained by the lead surgeon and the team. Virtual stimulators and wet labs should be provided by the hospital to the team to practice their first 8-10 cases until they gain expertise ---Continue

Design and feasibility study of a HCPS framework-based VR alpine skiing decision-making training system

Alpine skiing beginners are easily injured if they fail make the right decisions when faced with unexpected situations. Excellent decision-making requires extensive training, while alpine skiing training is limited by time and space. To address these issues, this paper combines VR and machine learning techniques to develop a high fidelity VR skiing decision-making training system based on HCPS framework. The aim is to train beginner skiers in decision-making by creating automated and repeatable virtual scenarios. Firstly, physical world (ski resort environment, and user motions) was mapped into the VR world using the HCPS framework. Afterwards, the user’s decision motions were made based on the scenarios provided, while being identified through an improved SVM (94.5% accuracy). The motion recognition results were fed into the expert system to assess the feasibility of the decision-making. Finally, the feasibility of the system was assessed with a combination of subjective and objective approaches. The results showed that the VR system significantly improved the participants’ skiing decision-making ability and speed compared to traditional 2D video training. The high fidelity environment reduced participants’ motion sickness symptoms and resulted in a better sensory experience. In addition, participants were more focused in the high fidelity environment. Overall, these results demonstrated the construct validity of the virtual ski decision-making training system. This study changes the traditional training method to create an immersive training environment, and provides an effective avenue for the development of sports training.

36‐1: 360‐degree Transparent Light Field Display with Highly‐Directional Holographic Screens for Fully Volumetric 3D Video Experience

We have developed a novel 360‐degree transparent light field display with 120 viewpoints for fully volumetric 3D video experience. It was achieved by a rotating cylindrical transparent highly‐directional holographic screen and a high‐frame‐rate projector. It enables multiple people to simultaneously view bright and occlusion‐capable volumetric images from any direction.

Multi-Stage Edge-Guided Stereo Feature Interaction Network for Stereoscopic Image Super-Resolution

Stereo image super-resolution (SR) aims to simultaneously increase the resolution of stereo image pairs, which benefits many downstream three-dimensional (3D) multimedia broadcasting and stereo vision-related tasks, such as 3D television broadcasting and stereo matching. A key insight in convolutional neural networks-based stereo image SR is to enforce stereo feature interactions between the two stereo views to explore complementary cross-view features that can facilitate the SR in both views. To fully exploit the cross-view stereo features, in this paper we propose a new multi-stage network, cascaded by several stereo feature interactions, progressively improving the SR quality from coarse to fine. In particular, an edge-guided stereo attention mechanism is proposed to be embedded into each stereo feature interaction to better capture consistent structure details of the cross-views. Followed by stereo feature fusion and reconstruction modules, we finally put together a multi-stage edge-guided stereo feature interaction network (MESFINet) for stereo image SR. Comprehensive experiments on KITTI2012, KITTI2015, Middlebury, and Flickr1024 benchmark datasets show that the proposed MESFINet achieves superior performance against the state-of-the-art stereo image SR methods and can be used to improve the accuracy of stereo matching.

Successful treatment of pelvic venous incompetence by a single-stage combined laparoscopic intervention: A case report

Introduction: Pelvic venous incompetence (PVI) is the most common chronic pathology, mainly affecting women of different ages. Its characteristic signs include pelvic venous fullness syndrome and chronic pelvic pain, contributing to temporary disability and impaired reproductive function. While there are known surgical treatment methods for pelvic pain in PVI, they have numerous complications. Case presentation: A 39-year-old female patient presented with PVI and chronic pelvic pain, dysmenorrhea, and infertility. They underwent simultaneous combined reconstructiveplastic conservative surgical treatment under 3D laparoscopic video monitoring. Conclusion: The patient's treatment outcome was characterized by decreased pelvic pain, normalized menstrual function, and a long-awaited uterine pregnancy ending with the birth of a full-term baby. The two successively performed surgical treatment procedures had a cumulative therapeutic effect, relieving the patient's varicose pelvic pain due to various causes and restoring their reproductive function.

The Deep Learning Generative Adversarial Random Neural Network in data marketplaces: The digital creative

Generative Adversarial Networks (GANs) have been proposed as a method to generate multiple replicas from an original version combining a Discriminator and a Generator. The main applications of GANs have been the casual generation of audio and video content. GANs, as a neural method that generates populations of individuals, have emulated genetic algorithms based on biologically inspired operators such as mutation, crossover and selection. This article presents the Deep Learning Generative Adversarial Random Neural Network (RNN) with the same features and functionality as a GAN. Furthermore, the presented algorithm is proposed for an application, the Digital Creative, that generates tradeable replicas in a Data Marketplace, such as 1D functions or audio, 2D and 3D images and video content. The RNN Generator creates individuals mapped from a latent space while the GAN Discriminator evaluates them based on the true data distribution. The performance of the Deep Learning Generative Adversarial RNN has been assessed against several input vectors with different dimensions, in addition to 1D functions and 2D images. The presented results are successful: the learning objective of the RNN Generator creates tradeable replicas at low error, whereas the RNN Discriminator learning target identifies unfit individuals.

Improved automatic identification of isolated rapid eye movement sleep behavior disorder with a 3D time-of-flight camera.

Automatic 3D video analysis of the lower body during rapid eye movement (REM) sleep has been recently proposed as a novel tool for identifying people with isolated REM sleep behavior disorder (iRBD), but, so far, it has not been validated on unseen subjects. This study aims at validating this technology in a large cohort and at improving its performances by also including an analysis of movements in the head, hands and upper body. Fifty-three people with iRBD and 128 people without RBD (of whom 89 had sleep disorders considered RBD differential diagnoses) were included in the study. An automatic algorithm identified movements from 3D videos during REM sleep in four regions of interest (ROIs): head, hands, upper body and lower body. The movements were divided into categories according to duration: short (0.1-2 s), medium (2-15 s) and long (15-300 s). For each ROI and duration range, features were obtained from the identified movements. Logistic regression models using as predictors the features from one single ROI or a combination of ROIs were trained and tested in a 10-runs 10-fold cross-validation scheme on the task of differentiating people with iRBD from people without RBD. The best differentiation was achieved using short movements in all four ROIs (test accuracy 0.866 ± 0.007, test F1 score = 0.783 ± 0.010). Single group analyses showed that people with iRBD were distinguished successfully from subjects with RBD differential diagnoses. Automatic 3D video analysis might be implemented in clinical routine as a supportive screening tool for identifying people with RBD.

From 2D to 3D video conferencing: modular RGB-D capture and reconstruction for interactive natural user representations in immersive extended reality (XR) communication

With recent advancements in Virtual Reality (VR) and Augmented Reality (AR) hardware, many new immersive Extended Reality (XR) applications and services arose. One challenge that remains is to solve the social isolation often felt in these extended reality experiences and to enable a natural multi-user communication with high Social Presence. While a multitude of solutions exist to address this issue with computer-generated “artificial” avatars (based on pre-rendered 3D models), this form of user representation might not be sufficient for conveying a sense of co-presence for many use cases. In particular, for personal communication (for example, with family, doctor, or sales representatives) or for applications requiring photorealistic rendering. One alternative solution is to capture users (and objects) with the help of RGBD sensors to allow real-time photorealistic representations of users. In this paper, we present a complete and modular RGBD capture application and outline the different steps needed to utilize RGBD as means of photorealistic 3D user representations. We outline different capture modalities, as well as individual functional processing blocks, with its advantages and disadvantages. We evaluate our approach in two ways, a technical evaluation of the operation of the different modules and two small-scale user evaluations within integrated applications. The integrated applications present the use of the modular RGBD capture in both augmented reality and virtual reality communication application use cases, tested in realistic real-world settings. Our examples show that the proposed modular capture and reconstruction pipeline allows for easy evaluation and extension of each step of the processing pipeline. Furthermore, it allows parallel code execution, keeping performance overhead and delay low. Finally, our proposed methods show that an integration of 3D photorealistic user representations into existing video communication transmission systems is feasible and allows for new immersive extended reality applications.

Super-resolution of 2D ultrasound images and videos

This paper proposes a novel deep-learning framework for super-resolution ultrasound images and videos in terms of spatial resolution and line reconstruction. To this end, we up-sample the acquired low-resolution image through a vision-based interpolation method; then, we train a learning-based model to improve the quality of the up-sampling. We qualitatively and quantitatively test our model on different anatomical districts (e.g., cardiac, obstetric) images and with different up-sampling resolutions (i.e., 2X, 4X). Our method improves the PSNR median value with respect to SOTA methods of 1.7% on obstetric 2X raw images, 6.1% on cardiac 2X raw images, and 4.4% on abdominal raw 4X images; it also improves the number of pixels with a low prediction error of 9.0% on obstetric 4X raw images, 5.2% on cardiac 4X raw images, and 6.2% on abdominal 4X raw images. The proposed method is then applied to the spatial super-resolution of 2D videos, by optimising the sampling of lines acquired by the probe in terms of the acquisition frequency. Our method specialises trained networks to predict the high-resolution target through the design of the network architecture and the loss function, taking into account the anatomical district and the up-sampling factor and exploiting a large ultrasound data set. The use of deep learning on large data sets overcomes the limitations of vision-based algorithms that are general and do not encode the characteristics of the data. Furthermore, the data set can be enriched with images selected by medical experts to further specialise the individual networks. Through learning and high-performance computing, the proposed super-resolution is specialised to different anatomical districts by training multiple networks. Furthermore, the computational demand is shifted to centralised hardware resources with a real-time execution of the network’s prediction on local devices.Graphical abstract

Virtual Forest Bathing Programming as Experienced by Disabled Adults with Mobility Impairments and/or Low Energy: A Qualitative Study

Background: Although access to nature is demonstrated to benefit health and wellbeing, adults with mobility impairments and/or low energy often face barriers in accessing nature environments and nature-based programs. This study aimed to examine the experiences and impacts of virtual forest bathing by capturing the perspectives of disabled adults with mobility impairments and/or low energy. Methods: A total of 26 adults with mobility impairments provided written and spoken qualitative feedback during and following virtual forest bathing programs and 23 participants provided feedback at a one month follow-up. Virtual programs were presented online, using an accessible format, 2D videos, and images of nature accompanied by guidance led by a certified forest bathing guide and mindfulness teacher. The programs involved disabled facilitators and participants, which created a social environment of peer support. Results: Qualitative thematic analysis revealed 10 themes comprising intervention themes (virtual delivery and soothing facilitation); process themes (nature connection, relaxation, embodiment, and memories with complex emotions); and outcome themes (happiness, agency, metaphor making, and belonging). Conclusions: Virtual forest bathing may offer an effective adjunct to improve wellbeing and provide peer support for disabled adults with mobility impairments and/or low energy.

The Classification of Movement in Infants for the Autonomous Monitoring of Neurological Development

Neurodevelopmental delay following extremely preterm birth or birth asphyxia is common but diagnosis is often delayed as early milder signs are not recognised by parents or clinicians. Early interventions have been shown to improve outcomes. Automation of diagnosis and monitoring of neurological disorders using non-invasive, cost effective methods within a patient’s home could improve accessibility to testing. Furthermore, said testing could be conducted over a longer period, enabling greater confidence in diagnoses, due to increased data availability. This work proposes a new method to assess the movements in children. Twelve parent and infant participants were recruited (children aged between 3 and 12 months). Approximately 25 min 2D video recordings of the infants organically playing with toys were captured. A combination of deep learning and 2D pose estimation algorithms were used to classify the movements in relation to the children’s dexterity and position when interacting with a toy. The results demonstrate the possibility of capturing and classifying children’s complexity of movements when interacting with toys as well as their posture. Such classifications and the movement features could assist practitioners to accurately diagnose impaired or delayed movement development in a timely fashion as well as facilitating treatment monitoring.

The effect of simulated natural environments in virtual reality and 2D video to reduce stress.

Stress is a common problem associated with poor physical and psychological health. Exposure to the natural environment is one method for reducing stress. The real and simulated natural environments have a restorative effect on stress reduction. In contrast to the real environment, simulated natural environments, such as virtual reality and 2D video, provide safer and more controllable exposure. Several studies on the restorative effects of the natural environment in virtual reality and 2D video have been conducted. However, the difference between the two in reducing stress must be clarified. This study was conducted to determine the effect of the simulated natural environments in virtual reality and 2D video and their differences in reducing stress. This study hypothesizes that both simulated natural environments in virtual reality and 2D video can reduce stress, but there is a difference between them in reducing stress. Fifty-three subjects were divided into two experimental groups: 2D video (n = 28) and virtual reality (n = 25). The results indicated that simulated natural environments in virtual reality and 2D video reduced stress. However, there was no difference between the two groups regarding stress reduction.

Multi-layer and Multi-scale feature aggregation for DIBR-Synthesized image quality assessment

Depth-Image-Based Rendering (DIBR) is one of the main fundamental techniques for generating new viewpoints in 3D video applications such as multi-viewpoint video (MVV), free viewpoint video (FVV) and virtual reality (VR). Due to the imperfections of color images, depth maps or texture restoration techniques, several types of distortions occur in synthesized views. However, most of related works evaluated the quality of DIBR-synthesized views by only detecting a specific type of distortion, such as stretching, black holes, blurring, etc., which were unable to accurately evaluate the quality of DIBR-synthesized views. In this paper, a new no-reference image quality assessment method is proposed to evaluate the quality of DIBR-synthesized images by combining multi-layer and multi-scale features of images. To be specific, the distortions introduced by different stages of virtual viewpoint synthesis are first analyzed, and then multi-layer and multi-scale features are extracted to estimate the degree of texture and structure distortions. As a result, individual quality scores associated with two types of distortions (e.g., structural distortion and texture distortion) are aggregated to an overall image quality. Experimental results on two publicly available DIBR datasets show that the method has better performance than the state-of-the-art models.Index Terms: image quality assessment, DIBR-synthesized image, distortion correction, BIQA.

Reliability and validity of 2D-video analysis to objectively assess hamstring performance during the H-test

ABSTRACT The H-test is commonly used during return-to-sport decisions after hamstring muscle injury. The primary aim was to evaluate the reliability of two-dimensional (2D) video analysis for the H-Test. The second aim was to assess its validity compared to an electronic gyroscope (gold standard), and the third aim was to establish normative values. We conducted a cross-sectional study including 30 healthy individuals. Mean, maximal velocities (VMean and Vmax) and range of motion (ROM) of hip flexion were captured during the H-test to evaluate inter-rater and test-retest reliability using intraclass correlation coefficient (ICC2,1) and standard error of measurement (SEM). Correlation analysis (r) and as typical error of estimate (TEE) were used to assess the validity between the video and the gyroscope. Reliability was excellent for ROM (ICC:0.91, [95% CI:0.83–0.95]), moderate for VMean (ICC:0.57; [95% CI:0.32–0.74]) and VMax (ICC:0.64, [95% CI:0.43–0.79]). Strong positive correlations were found between video and gyroscope for VMean (r = 0.79, [95% CI:0.71–0.86]) and VMax (r = 0.84, [95% CI:0.77–0.89]) and very strong for ROM (r = 0.89, [95% CI:0.85–0.93]). Males exhibited a higher VMax (p < 0.001) than females, while females had a higher ROM (p < 0.001). 2D-video analysis is a valid and reliable method to assess ROM during the H-Test and could easily be implemented in clinical practice.

Revisiting Raindrop Axis Ratios Based on 3D Oblate Spheroidal Reconstruction: 2D Video Disdrometer Observations During Tropical Cyclone Passages

AbstractRaindrops are usually parameterized by oblate spheroids with axis ratios (ARs) determined from projected images. However, the adequacy of representing the 3‐Dimensional (3D) nature of raindrops with their 2‐Dimensional (2D) shadows remains an open question. This study reexamines raindrop AR parameterizations by applying a 3D oblate spheroidal reconstruction algorithm to 2D video disdrometer (2DVD) observations during the passages of 14 tropical cyclones. The results reveal an appreciable AR overestimation for large raindrops when retrieved from a single camera and especially for 2DVD raw products. This overestimation propagates into the simulated differential reflectivity (ZDR), resulting in an underestimation of ZDR by up to 0.5 dB at X‐band when large raindrops dominate the raindrop size distribution (RSD), which would introduce an underestimation of more than 20% for heavy rain rate (&gt;60 mm hr−1) estimation by polarimetric radar. In precipitating systems with drastically changing RSDs, the importance of adopting more realistic ARs is warranted.

Community and domestic violence are associated with DNA methylation GrimAge acceleration and heart rate variability in adolescents

ABSTRACT Background: Cumulative exposure to violence can change the regulation of epigenetic and physiological markers. Although violence has been associated with accelerated cellular aging, little is known about associations with cardiac autonomic activity. Objective: The current study aimed to investigate the relationship of exposure to community and domestic violence (CDV) with vagal activity and epigenetic aging acceleration. Methods: A total of 86 adolescents (57% female) were evaluated and interviewed at two time-points in São Gonçalo (2014–2019), a Brazilian city with high levels of violence. Exposure to CDV was assessed in both time-points. GrimAge acceleration was calculated from saliva DNA methylation using Infinium HumanMethylation450K (Illumina) collected in the first assessment. Heart rate variability (HRV) was collected during two stress tasks at the second assessment. Results: The exposure to violence witnessed or directly experienced at home and in the community increased significantly (t = 4.87, p < .01) across two-time points, and males had reported higher violence exposure (t = 2.06, p = .043). Violence at 1st assessment was significantly associated with GrimAge acceleration (B = .039, p value = .043). Violence at both assessments were associated with HRV measured during the narration of the worst trauma (traumaHRV) (B = .009, p value = .039, and B = .007, p value = .024, 1st and 2nd assessment respectively). GrimAge acceleration was significantly associated with traumaHRV (B = .043, p value = .049), and HRV measured during a 3D roller coaster video (B = .061, p value = .024). Conclusions: We found relevant evidence that experiencing violence during adolescence is associated with epigenetic aging and stress-related vagal activity. Understanding these factors during this period could contribute to the development of early interventions for health promotion. HIGHLIGHTS Higher exposure to Community and domestic violence is associated with increased GrimAge acceleration. Higher GrimAge acceleration is associated with increased stress-related vagal activity. Exposure to community and domestic violence increased significantly over time.

Future of virtual education and telementoring

To summarize recent technological advancements in medical and surgical education and explore what the future of medicine might be as it relates to blockchain technology, the metaverse, and web3. Through the use of digitally assisted ophthalmic surgery and high dynamic range 3D cameras, it is now possible to record and live stream 3D video content. Although the 'metaverse' is still in its early stages, there are a variety of proto-metaverse technologies that exist to facilitate user interactions that can mimic the real world through the use of shared digital environments and 3D spatial audio. Advanced blockchain technologies can allow for further development of interoperable virtual worlds where a user has an on-chain identity, credentials, data, assets, and much more that they can carry across platforms seamlessly. As remote real-time communication becomes an integral part of human interaction, 3D live streaming has the potential to revolutionize ophthalmic education by removing traditional geographic and physical constraints of in-person surgical viewing. The incorporation of metaverse and web3 technologies has created new outlets for knowledge sharing that may improve how we operate, teach, learn, and transfer knowledge.

PD-0906 Preventing potential treatment collisions on a TrueBeam system using a 3D video game engine

PD-0906 Preventing potential treatment collisions on a TrueBeam system using a 3D video game engine