3D Environment Research Articles

Metaverse-based cardiac magnetic resonance simulation to overcome claustrophobia (META-MRI): a feasibility trial

Abstract Background Cardiac Magnetic Resonance Imaging (CMR) is crucial for diagnosing and stratifying cardiac disease, but up to 10% of patients face challenges due to claustrophobia. Conventional methods for overcoming claustrophobia have limited success. The metaverse, an immersive 3D environment using virtual and augmented reality, is under investigation for healthcare applications. A metaverse-based VR CMR simulation program was developed to assist patients with claustrophobia, but its clinical effectiveness is unexplored. Methods The META-MRI study involved 450 patients screened, with 32 enrolled. They were randomly assigned to intervention (Metaverse-based simulation + standard care) or control (standard care alone). The Metaverse program used a VR headset within the Aimedis metaverse platform. Training sessions occurred before scheduled CMR exams, with anxiolytic treatment when necessary. Primary endpoints included successful CMR completion, anxiety assessment using the State-Trait Anxiety Inventory (STAI), and usability through a questionnaire. Results Patients in the intervention group (n=16) completed Metaverse training successfully. During simulations, anxiety decreased (STAI scores: 73 ± 8 to 51 ± 11, p=0.02). Intervention group completion rate was significantly higher (81%) than the control group (37%) during real CMR exams (p=0.01). Usability scores indicated moderate satisfaction. Discussion The metaverse-based program significantly improved CMR completion rates and reduced anxiety levels in patients with severe claustrophobia compared to standard care alone. Traditional methods for claustrophobia have limitations, making the metaverse a promising tool. Despite promising results, further research is necessary, considering identified usability issues and economic implications. Conclusion This feasibility trial demonstrates the effectiveness of a metaverse-based simulation program in enhancing compliance with CMR exams for patients with severe claustrophobia. The strategy, combined with standard care, led to reduced anxiety and increased CMR completion. Further investigation with larger cohorts is required to validate these findings, addressing usability concerns and assessing the cost-effectiveness of incorporating the metaverse into routine clinical practice.Clinical and demographic characteristics

Human amniotic membrane scaffold enhances adipose mesenchymal stromal cell mitochondrial bioenergetics promoting their regenerative capacities.

The human amniotic membrane (hAM) has been applied as a scaffold in tissue engineering to sustain stem cells and enhance their regenerative capacities. We investigated the molecular and biochemical regulations of mesenchymal stromal cells (MSCs) cultured on hAM scaffold in a three-dimensional (3D) setting. Culture of adipose-MSCs (AMSCs) on decellularized hAM showed significant improvement in their viability, proliferative capacity, resistance to apoptosis, and enhanced MSC markers expression. These cultured MSCs displayed altered expression of markers associated with pro-angiogenesis and inflammation and demonstrated increased potential for differentiation into adipogenic and osteogenic lineages. The hAM scaffold modulated cellular respiration by upregulating glycolysis in MSCs as evidenced by increased glucose consumption, cellular pyruvate and lactate production, and upregulation of glycolysis markers. These metabolic changes modulated mitochondrial oxidative phosphorylation (OXPHOS) and altered the production of reactive oxygen species (ROS), expression of OXPHOS markers, and total antioxidant capacity. They also significantly boosted the urea cycle and altered the mitochondrial ultrastructure. Similar findings were observed in bone marrow-derived MSCs (BMSCs). Live cell imaging of BMSCs cultured in the same 3D environment revealed dynamic changes in cellular activity and interactions with its niche. These findings provide evidence for the favorable properties of hAM as a biomimetic scaffold for enhancing the in vitro functionality of MSCs and supporting their potential usefulness in clinical applications.

AEFF-SSC: an attention-enhanced feature fusion for 3D semantic scene completion

Abstract Three-dimensional (3D) occupancy perception technology aims to enable autonomous vehicles to observe and understand dense 3D environments. Estimating the complete geometry and semantics of a scene solely from visual images is challenging. However, humans can easily conceive the complete form of objects based on partial key information and their own experience. This ability is crucial for recognizing and interpreting the surrounding environment. To equip 3D occupancy perception systems with a similar capability, a 3D semantic scene completion method called AEFF-SSC is proposed. This method deeply explores boundary and multi-scale information in voxels, aiming to reconstruct 3D geometry more accurately. We have specifically designed an attention-enhanced feature fusion module that effectively fuses image feature information from different scales and focuses on feature boundary information, thereby more efficiently extracting voxel features. Additionally, we introduce a semantic segmentation module driven by a 3D attention-UNet network. This module combines a 3D U-Net network with a 3D attention mechanism. Through feature fusion and feature weighting, it aids in restoring 3D spatial information and significantly improves the accuracy of segmentation results. Experimental verification on the SemanticKITTI dataset demonstrates that AEFF-SSC significantly outperforms other existing methods in terms of both geometry and semantics. Specifically, within a 12.8 m × 12.8 m area ahead, our geometric occupancy accuracy has achieved a significant improvement of 71.58%, and at the same time, the semantic segmentation accuracy has also increased remarkably by 54.20%.

Assessing the Effects of Sensory Modality Conditions on Object Retention across Virtual Reality and Projected Surface Display Environments

Haptic feedback reportedly enhances human interaction with 3D data, particularly improving the retention of mental representations of digital objects in immersive settings. However, the effectiveness of visuohaptic integration in promoting object retention across different display environments remains underexplored. Our study extends previous research on the retention effects of haptics from virtual reality to a projected surface display to assess whether earlier findings generalize to 2D environments. Participants performed a delayed match-to-sample task incorporating visual, haptic, and visuohaptic sensory feedback within a projected surface display environment. We compared error rates and response times across these sensory modalities and display environments. Our results reveal that visuohaptic integration significantly enhances object retention on projected surfaces, benefiting task performance across display environments. Our findings suggest that haptics can improve object retention without requiring fully immersive setups, offering insights for the design of interactive systems that assist professionals who rely on precise mental representations of digital objects.

Point cloud segmentation method based on an image mask and its application verification

Abstract Accurately perceiving three-dimensional (3D) environments or objects is crucial for the advancement of artificial intelligence (AI) interaction technologies. Currently, various types of sensors are employed to obtain point cloud data for 3D object detection or segmentation tasks. While this multi-sensor approach provides more precise 3D data than monocular or stereo cameras, it is also more expensive. The advent of RGB-D cameras, which provide both RGB images and depth information, addresses this issue.
In this study, we propose a point cloud segmentation method based on image masks. By using an RGB-D camera to capture color and depth images, we generate image masks through object recognition and segmentation. Given the mapping relationship between RGB image pixels and point clouds, these image masks can be further used to extract the point cloud data of the target objects. The experimental results revealed that the average accuracy of target segmentation was 84.78%, which was close to that of PointNet++. Compared with three traditional segmentation algorithms, the accuracy was improved by nearly 23.97%. The running time of our algorithm is reduced by 95.76% compared to the PointNet++ algorithm, which has the longest running time; and by 15.65% compared to the LCCP algorithm, which has the shortest running time among traditional methods. Compared with PointNet++, the segmentation accuracy was improved. This method addressed the issues of low robustness and excessive reliance on manual feature extraction in traditional point cloud segmentation methods, providing valuable support and reference for the accurate segmentation of 3D point clouds.

Classification of EEG evoked in 2D and 3D virtual reality: traditional machine learning vs. deep learning.

Virtual reality (VR) simulates real-life events and scenarios, widely used in education, entertainment, and medicine. VR can be presented in two or three dimensions (2D or 3D), and 3D VR produces a more realistic and immersive experience. Previous research has revealed that the electroencephalogram (EEG) induced by 3D VR has a different profile from that of 2D VR, manifesting in many aspects, such as the power of brain rhythm, brain activation, and brain functional connectivity. However, studies on how to classify EEG in 2D and 3D VR were limited. 64-channel EEG was recorded, while visual stimuli were given in 2D and 3D VR. The classification of these recorded EEG signals was done using two machine learning methods: the traditional method and the deep learning method. In the traditional machine learning classification, EEG features of power spectral density (PSD) and common spatial patterns (CSP) were extracted, and three classification algorithms, support vector machines (SVM), K-nearest neighbors (KNN), and random forests (RF), were used. A specialized convolutional neural network, EEGNet, was used in the deep learning classification. These classification approaches were compared with respect to their classification performance. In aspects of four performance evaluations for classification, accuracy, precision, recall, and F1-score, respectively, classification using the deep learning method is better than the traditional machine learning approaches. Classification accuracy with deep learning with EEGNet could reach up to 97.86%. The classification performance of 2D and 3D VR-induced EEG can be achieved with EEGNet-based deep learning, outperforming conventional machine learning approaches. Given the role of EEGNet, which is designed for EEG-based brain-computer interfaces (BCI), better performance classification of EEG in 2D and 3D VR environments might be predicted to be helpful for the application of 3D VR in BCI.

Position and reduced attitude trajectory tracking control of quadrotors: Theory and experiments

Multirotor aerial vehicles are versatile flying robots that perform hovering, vertical take-off and landing, and aggressive maneuvers in a 3D environment. Due to their underactuated nature, the aerial vehicles' position and orientation cannot be controlled independently. For this reason, most of the quadrotors' tasks involved position tracking or regulation tasks. This paper focuses on the position-tracking problem of quadrotors using the reduced orientation of the vehicle, meaning that only two degrees of freedom of the robot's orientation are controlled. We propose an almost global exponential reduced attitude control law that aligns the aerial robot's thrust direction with the desired force that drives the robot along the desired position trajectory. For the translational subsystem, we propose a dynamic control law that drives the position and velocity of the quadrotors asymptotically to the desired trajectories. The proposed attitude control law is computationally simple, and thus, it is suitable to run on board. Finally, we provide experimental results performed on a low-cost quadrotor and a comparison study with a full-attitude controller to illustrate the performance and advantages of the proposed control laws.

Conceptual Model of Graph-based Individual Tree and Its Utilization in Digital Twin and Metaverse of Urban Forest

Abstract. 3D city models are an important cornerstone in the development of digital twin cities, allowing for various analyses and simulations. CityGML, as an open standard for 3D city models, emphasizes five main aspects: scale or level of detail (LoD), semantics, geometry, topology, and appearance. One of the important elements in CityGML 3D city models is representation of vegetation. Vegetation in CityGML, especially individual trees, is still limited in terms of detail and resolution, reducing its effectiveness for specific applications. This paper proposes a graph-based conceptual model for individual trees that conforms to the CityGML v2.0 specification. The model refers to the morphological structure of a tree consisting of roots, trunk, and crown (including branches, twigs, and leaves), and considers the five main aspects of CityGML. By enhancing semantic information and geometric details, the model aims to provide an information-rich and realistic representation of individual trees in a 3D city environment. The 3D tree model created based on this conceptual model may be applicable in the development of digital twin urban forests and virtual forest simulations that utilize immersive technologies such as the metaverse. This research opens up new opportunities in the development of solitary vegetation objects through CityGML ADE by including more detailed semantic and topological information.

3D Voxel-based Path Planning for AVs in Dynamic Complex Environments

Abstract. With the rapid advancement of science and technology, autonomous driving technology has been widely applied in various fields such as the automotive industry, intelligent warehousing, and emergency rescue. Traditional 2D path planning has limitations in dealing with the complex and dynamic environments of the real world, making it difficult to meet the high precision and complex path planning requirements of autonomous vehicles. With the development of data acquisition technology, large-scale point cloud data has become easily accessible, which can better extract the three-dimensional complex of the environment. Nevertheless, the unstructured nature and vast amount of point cloud data make path planning on it extremely challenging. Voxel models can effectively compress point cloud data and provide neighboring topological structures. In view of this, we propose a voxel-based framework for the path planning of autonomous vehicles in dynamic and complex 3D environments, which involves converting point cloud data into voxels and extracting navigation spaces. Subsequently, deep reinforcement learning techniques are utilized to achieve obstacle avoidance and navigation on the voxel map. It is expected that this framework will contribute to the development of 3D path planning and enhance the utilization of point cloud data in the navigation field.

Establishing Patient-Derived Xenograft (PDX) Models of Lymphomas.

Patient-derived xenograft (PDX) models of lymphoma typically involve the injection of human tumor cells into an immunocompromised murine host. PDXs have the advantage that the tumor cells grow in a 3D environment within the mouse, meaning the selection pressure of in vitro establishment is avoided and the tumor cells better maintain their genetic heterogeneity. Here, we outline a method for producing and maintaining a PDX model of lymphoma. We describe three different methods to isolate a single cell suspension of the primary patient tumor, followed by either subcutaneous or intraperitoneal injection into an immunocompromised mouse. We then detail how to monitor tumor growth and how to harvest, passage, and store the tumors once they have grown. We highlight and discuss important protocol considerations including technical hints as well as the advantages and disadvantages of the methods described.

Why do bats fly into cave doors? Inattentional blindness in echolocating animals.

Echolocating bats can navigate complex 3D environments by integrating prior knowledge of spatial layouts and real-time sensory cues. This study demonstrates that inattentional blindness to sensory information undermines successful navigation in Egyptian fruit bats, Rousettus aegyptiacus , a species that has access to vision and echolocation to traverse natural environments. Bats flew over repeated trials to a perch at a fixed location in the light, allowing them to navigate using both vision and echolocation. The experiment was then repeated in the dark to exclude the bat's use of vision. The perch was subsequently displaced by either 15 or 30 cm in one of six different directions (up, down, left, right, front, back). Echolocation behavior was recorded using a 25-channel microphone array, while flight paths were tracked using 13 motion capture cameras. The directional aim of echolocation clicks served as a metric for the bat's spatial attention to locations in their environment. In the light, bats modified their flight paths to successfully land on a perch that was moved 15 cm but surprisingly, often failed to land on it when displaced by 30 cm. In the dark, bats often failed to land on the perch after it was moved by only 15 cm. Landing failures suggest that learned spatial priors invoked inattentional blindness to changes in the environment, which interfered with successful navigation. In both the light and dark, when bats failed to land on the perch at its new location, they directed their attention toward the original perch position. Performance differences in the light and dark suggest that the bat's attentional spotlight may be narrower when it relies on echolocation than vision. To our knowledge, these findings provide the first evidence of inattentional blindness in a flying echolocating animal, demonstrating that spatial priors can dominate sensory processing during navigation.

Spatially-Resolved Organoid Transfection by Porous Silicon-Mediated Optoporation.

Engineering the spatial organisation of organotypic cultures is pivotal for refining tissue models that are useful for gaining deeper insights into complex, non-cell autonomous processes. These advanced models are key to improving the understanding of fundamental biological mechanisms and therapeutic strategies. Controlling gene regulation through spatially-resolved delivery of nucleic acids provides an attractive approach to produce such tissue models. An emerging strategy for spatially-resolved transfection uses photosensitizing nanoparticles coupled with laser pulses to optoporate cells in culture and locally mediate gene delivery. However, localized optoporation in 3D systems remains challenging. Here we propose a solution to this longstanding hurdle, demonstrating that porous silicon nanoparticles are a safe and bioresorbable photosensitising nanomaterial capable of spatially-resolved transfection of mRNA in MCF-7 organoids by near-infrared two-photon optoporation. Functionalization with an azobenzene-lysine photo-switchable moiety enhances the transfection efficiency of the nanoparticles up to 84% in a 2D cell system. Moreover, the nanoparticles enable spatially selective mRNA transfection to MCF-7 spheroids, demonstrating targeted gene delivery in complex 3D cellular environments. The approach for spatially-resolved 3D optoporation offers a way forward for the design of tailored spheroids and organoids by spatially selective nucleic acids delivery.

Prediction of Attention Groups and Big Five Personality Traits from Gaze Features Collected from an Outlier Search Game.

This study explores the intersection of personality, attention and task performance in traditional 2D and immersive virtual reality (VR) environments. A visual search task was developed that required participants to find anomalous images embedded in normal background images in 3D space. Experiments were conducted with 30 subjects who performed the task in 2D and VR environments while their eye movements were tracked. Following an exploratory correlation analysis, we applied machine learning techniques to investigate the predictive power of gaze features on human data derived from different data collection methods. Our proposed methodology consists of a pipeline of steps for extracting fixation and saccade features from raw gaze data and training machine learning models to classify the Big Five personality traits and attention-related processing speed/accuracy levels computed from the Group Bourdon test. The models achieved above-chance predictive performance in both 2D and VR settings despite visually complex 3D stimuli. We also explored further relationships between task performance, personality traits and attention characteristics.

WISH-BONE: Whole-mount insitu histology, to label osteocyte mRNA and protein in 3D adult mouse bones.

Bone is a three-dimensional (3D) highly dynamic tissue under constant remodeling. Commonly used tools to investigate bone biology require sample digestion for biomolecule extraction or provide only two-dimensional (2D) spatial information. There is a need for 3D tools to investigate spatially preserved biomarker expression in osteocytes. In this work, we present a new method, WISH-BONE, to label osteocyte messenger RNA (mRNA) and protein in whole-mount mouse bone. For mRNA labeling, we used hybridization chain reaction-fluorescence insitu hybridization (HCR-FISH) to label genes of interest in osteocytes. For protein labeling, samples were preserved using an epoxy-based solution that protects tissue structure and biomolecular components. Then an enzymatic matrix permeabilization step was performed to enable antibody penetration. Immunostaining was used to label various proteins involved in bone homeostasis. We also demonstrate the use of customized fluorescent nanobodies to target and label proteins in the cortical bone (CB). However, the relatively dim signal observed from nanobodies' staining limited detection. mRNA and protein labeling were performed in separate samples. In this study, we share protocols, highlight opportunities, and identify the challenges of this novel 3D labeling method. They are the first protocols for whole-mount osteocyte 3D labeling of mRNA and protein in mature mouse bones. WISH-BONE will allow the investigation of molecular signaling in bone cells in their 3D environment and could be applied to various bone-related fields of research.

Mutant p53 regulates cancer cell invasion in complex three-dimensional environments through mevalonate pathway-dependent Rho/ROCK signaling.

This study addresses the contribution of mutant p53 to the process of cancer cell dissemination in physiologically relevant three-dimensional environments - a key characteristic of metastatic disease. Several mutant p53 proteins display pro-oncogenic activity with respect to cancer cell invasion in 3D environments via mevalonate pathway-dependent Rho/ROCK signaling axis.

Energy Metabolism Behavior and Response to Microenvironmental Factors of the Experimental Cancer Cell Models Differ From That of Actual Human Tumors.

Analysis of the biochemical differences in the energy metabolism among bi-dimensional (2D) and tri-dimensional (3D) cultured cancer cell models and actual human tumors was undertaken. In 2D cancer cells, the oxidative phosphorylation (OxPhos) fluxes range is 2.5-19 nmol O2/min/mg cellular protein. Hypoxia drastically decreased OxPhos flux by 2-3 times in 2D models, similar to what occurs in mature multicellular tumor spheroids (MCTS), a representative 3D cancer cell model. However, mitochondrial protein contents and enzyme activities were significantly different between both models. Moreover, glycolytic fluxes were also significantly different between 2D and MCTS. The glycolytic flux range in 2D models is 1-34 nmol lactate/min/mg cellular protein, whereas in MCTS the range of glycolysis fluxes is 60-80 nmol lactate/min/mg cellular. In addition, sensitivity to anticancer canonical and metabolic drugs was greater in MCTS than in 2D. Actual solid human tumor samples show lower (1.6-4.5 times) OxPhos fluxes compared to normoxic 2D cancer cell cultures. These observations indicate that tridimensional organization provides a unique microenvironment affecting tumor physiology, which has not been so far faithfully reproduced by the 2D environment. Thus, the analysis of the resemblances and differences among cancer cell models undertaken in the present study raises caution on the interpretation of results derived from 2D cultured cancer cells when they are extended to clinical settings. It also raises awareness about detecting which biological and environmental factors are missing in 2D and 3D cancer cell models to be able to reproduce the actual human tumor behavior.

A guide to studying 3D genome structure and dynamics in the kidney.

The human genome is tightly packed into the 3D environment of the cell nucleus. Rapidly evolving and sophisticated methods of mapping 3D genome architecture have shed light on fundamental principles of genome organization and gene regulation. The genome is physically organized on different scales, from individual genes to entire chromosomes. Nuclear landmarks such as the nuclear envelope and nucleoli have important roles in compartmentalizing the genome within the nucleus. Genome activity (for example, gene transcription) is also functionally partitioned within this 3D organization. Rather than being static, the 3D organization of the genome is tightly regulated over various time scales. These dynamic changes in genome structure over time represent the fourth dimension of the genome. Innovative methods have been used to map the dynamic regulation of genome structure during important cellular processes including organism development, responses to stimuli, cell division and senescence. Furthermore, disruptions to the 4D genome have been linked to various diseases, including of the kidney. As tools and approaches to studying the 4D genome become more readily available, future studies that apply these methods to study kidney biology will provide insights into kidney function in health and disease.

3D Scaffold-Based Culture System Enhances Preclinical Evaluation of Natural Killer Cell Therapy in A549 Lung Cancer Cells.

Cell-based immunotherapies have emerged as promising cancer treatment modalities, demonstrating remarkable clinical efficacy. As interest in applying immune cell-based therapies to solid tumors has gained momentum, experimental models that enable long-term monitoring and mimic clinical administration are increasingly necessary. This study explores the potential of scaffold-based cell culture technologies, specifically three-dimensional (3D) extracellular matrix (ECM)-like frameworks, as promising solutions. These frameworks facilitate unhindered immune cell growth and enable continuous cancer cell culture. The three-dimensional (3D) cell culture model was developed using tailored scaffolds for natural killer (NK) cell culture. Within this framework, A549 lung cancer cells were cocultured with NK cells, allowing real-time monitoring for up to 28 days. The expression of critical markers associated with anticancer drug resistance and epithelial-mesenchymal transition (EMT) was evaluated in cancer cells within this 3D culture context. Compared to conventional 2D monolayer cultures, this 3D scaffold-based culture revealed that solid tumor cells, specifically A549 cells, exhibited heightened resistance to anticancer drugs. Additionally, the 3D culture environment upregulated the expression of EMT markers namely vimentin, N-cadherin, and fibronectin, while NK and zEGFR-CAR-NK cells displayed anticancer effects. In the two-dimensional (2D) coculture, only zEGFR-CAR-NK cells exhibited such effects in the 3D coculture system, highlighting an intriguing inconsistency with the 2D culture model, further confirmed by in vivo experiments. This in vitro 3D cell culture model reliably predicts outcomes in NK immunotherapy experiments. Thus, it represents a valuable tool for investigating drug resistance mechanisms and assessing the efficacy of immune cell-based therapies. By bridging the gap between in vitro and in vivo investigations, this model effectively translates potential treatments into animal models and facilitates rigorous preclinical evaluations.

Auranofin loaded silk fibroin nanoparticles for colorectal cancer treatment.

Colorectal cancer (CRC) is the second most common cause of cancer related deaths worldwide and the prevalence in young people especially is increasing annually. In the search for innovative approaches to treat the disease, drug delivery systems (DDS) are promising owing to their unique properties, which allow improved therapeutic results with lower drug concentrations, overcoming drug resistance and at the same time potentially reducing side effects. Silk fibroin is a biopolymer that can be processed to obtain biocompatible and biodegradable nanoparticles that can be efficiently loaded by surface adsorption with small-molecule therapeutics and allow their transport and sustained release by modulating their pharmacokinetics. Auranofin (AF) has recently been repurposed for its strong anticancer activity and is currently in clinical trials. Its mechanism of action is through the inhibition of thioredoxin reductase enzymes, which play an essential role in several intracellular processes and are overexpressed in some tumours. Taking into account that AF has a low solubility in water, we propose silk fibroin nanoparticles (SFN) as AF carrier in order to improve its bioavailability, increasing cellular absorption and preventing its degradation or avoiding some resistance mechanisms. Here we report the preparation and characterization of a new formulation of AF-loaded silk fibroin nanoparticles (SFN-AF), its functionalization with FITC for the analysis of cellular uptake, as well as its cytotoxic activity against cell lines of human colorectal cancer (HT29 and HCT116) in both 2D and 3D cell cultures. 3D spheroid models provide a 3D environment which mimics the 3D aspects of CRC observed in vivo and represents an effective 3D environment to screen therapeutics for the treatment of CRC. The loaded nanoparticles showed a spherical morphology with a hydrodynamic diameter of ~ 160nm and good stability in aqueous solution due to their negative surface charges. FESEM-EDX analysis revealed a homogeneous distribution of Au clusters with high electron density on the surface of the nanoparticles. SFN-AF incubated in phosphate buffer at 37°C released 77% of the loaded AF over 10 days, showing an initial burst and then sustained release. Flow cytometry analysis showed that FITC-SFN-AF was efficiently internalized by both cell lines, which was confirmed by confocal microscopy imaging. SFN enhanced the cytotoxicity of AF in 2D cultures in both CRC lines. Promising results were also obtained in 3D culture paving the way for future application of this strategy as a therapy for CRC.

Air Canvas Using MediaPipe for Computer Vision in Unity 3D Hand Tracking

In this paper, we present the implementation of an Air Canvas using MediaPipe for Computer Vision within a 3D environment using the Unity Game Engine. In our previous work, we found that, regardless of the initial parameters, simulations often led to rapid extinctions. In this model, we implemented an Air Canvas and Computer Vision system integrated with Unity’s 3D World. Our goal was to achieve system stabilization, long-term operation, and more realistic simulation by incorporating 3D evolution. Using the Unity Game Engine, we created and managed a closed 3D ecosystem environment, either based on artificial or real-world maps. This simulation of ecosystems and the analysis of the data generated can serve as a starting point for further research, particularly in sustainability. Our system is openly accessible, allowing users to customize and upload their parameters, maps, and objects, and define inheritance and behavioural patterns, enabling them to test their hypotheses based on the data generated. The goal of this article is not to create and validate a model but to provide an IT tool. For evolutionary researchers, the system allows the creation and presentation of simulations, including animated conference presentations for enhanced visualization and engagement. The use of 3D simulation is particularly valuable for educational purposes, engaging students and increasing their interest in 3D interactive worlds. Students can observe how ecosystems behave, how natural selection supports adaptability, and how competition impacts species.