3D World Research Articles

When New Experience Leads to New Knowledge: A Computational Framework for Formalizing Epistemically Transformative Experiences.

The discovery of a new kind of experience can teach an agent what that kind of experience is like. Such a discovery can be epistemically transformative, teaching an agent something they could not have learned without having that kind of experience. However, learning something new does not always require new experience. In some cases, an agent can merely expand their existing knowledge using, e.g., inference or imagination that draws on prior knowledge. We present a computational framework, grounded in the language of partially observable Markov Decision Processes (POMDPs), to formalize this distinction. We propose that epistemically transformative experiences leave a measurable "signature" distinguishing them from experiences that are not epistemically transformative. For epistemically transformative experiences, learning in a new environment may be comparable to "learning from scratch" (since prior knowledge has become obsolete). In contrast, for experiences that are not transformative, learning in a new environment can be facilitated by prior knowledge of that same kind (since new knowledge can be built upon the old). We demonstrate this in a synthetic experiment inspired by Edwin Abbott's Flatland, where an agent learns to navigate a 2D world and is subsequently transferred either to a 3D world (epistemically transformative change) or to an expanded 2D world (epistemically non-transformative change). Beyond the contribution to understanding epistemic change, our work shows how tools in computational cognitive science can formalize and evaluate philosophical intuitions in new ways.

Implementation of an indoor optical camera communication and localization fusion system using infrared LED markers

Recently, with the development of communication and positioning technologies, progressively higher demands have been placed on the timeliness of indoor communication and the effectiveness of indoor positioning. This paper proposes a low-cost and full-time domain coverage indoor communication and localization fusion system that uses an infrared camera to identify mobile LEDs based on region of interest optical camera communication (RoI-OCC) and calculate three-dimensional (3D) world coordinates based on perspective-n-point (PnP) algorithm. An intermediate delimiter modulation-demodulation method is designed to achieve asynchronous OCC. In processing infrared images, a region-growth-based absolute-directional-mean-difference (RG-ADMD) algorithm is developed to detect infrared targets and recover the scale. Additionally, a local-contrast-method-based Lucas-Kanade (LCM-LK) optical flow estimation algorithm is designed to track infrared targets and determine the centroid coordinates of pixels. In a 5×5 m experimental area, the established system can achieve a target detection recall rate of over 85%, the asynchronous OCC with error-free transmission and an average 3D positioning accuracy of 2.01 cm under a LCM-LK tracking algorithm.

An estimation method for vision-based autonomous landing system for fixed wing aircraft

In this paper, we propose a vision-based estimation method for autonomous landing of fixed-wing aircraft for Advanced Air Mobility. The concept of autonomous flight with minimal human intervention has gained significant interest with a particular focus on autonomous landing. The goal is to overcome the limitations of existing instrument landing systems and reduce reliance on GNSS during aircraft landing. The proposed system leverages the following techniques for high-performance and real-time operations in various real-world environments and during all stages of landing. A deep learning segmentation model was directly learned, created, and used for robust runway recognition performance against environmental changes around the runway. Algorithms were designed to adapt to different flight stages considering the varying information offered by image data when the aircraft is in mid-air and on the ground. The relative lateral position from the aircraft to the runway was estimated using bird’s-eye-view conversion and inverse projection techniques instead of conventional perspective-n-point methods for reducing the recognition error occurring from the conversion between 2D image and 3D world. Finally, the mixed-precision technique was used for the segmentation model to improve inference speed and ensure real-time operation in real-world deployment. Estimation stability and reliability were improved by using a Kalman Filter to cope with sensor uncertainties caused by the real-world flight environment. Consequently, we show that the average error in the lateral position estimation by the proposed method is comparable to the accuracy of a single GNSS and demonstrate successful autonomous landing of our test aircraft with a set of flight tests.

Air Canvas Using MediaPipe for Computer Vision in Unity 3D Hand Tracking

In this paper, we present the implementation of an Air Canvas using MediaPipe for Computer Vision within a 3D environment using the Unity Game Engine. In our previous work, we found that, regardless of the initial parameters, simulations often led to rapid extinctions. In this model, we implemented an Air Canvas and Computer Vision system integrated with Unity’s 3D World. Our goal was to achieve system stabilization, long-term operation, and more realistic simulation by incorporating 3D evolution. Using the Unity Game Engine, we created and managed a closed 3D ecosystem environment, either based on artificial or real-world maps. This simulation of ecosystems and the analysis of the data generated can serve as a starting point for further research, particularly in sustainability. Our system is openly accessible, allowing users to customize and upload their parameters, maps, and objects, and define inheritance and behavioural patterns, enabling them to test their hypotheses based on the data generated. The goal of this article is not to create and validate a model but to provide an IT tool. For evolutionary researchers, the system allows the creation and presentation of simulations, including animated conference presentations for enhanced visualization and engagement. The use of 3D simulation is particularly valuable for educational purposes, engaging students and increasing their interest in 3D interactive worlds. Students can observe how ecosystems behave, how natural selection supports adaptability, and how competition impacts species.

Virtual Reality (VR) Method to Improve Sense of Place for Interior Design Studio Students

Virtual reality (VR) technology has emerged in response to recent developments in the 3-dimensional (3D) world. VR enables people to engage in various metaverse world experiences in a more immersive way. Immersive learning is a learning method that uses 3D digital technology to facilitate the learning process by visualization in the classroom. This research used a case study of the Interior Design II studio course taken by level-2 students of the Department of Interior Design, School of Creative Industries, Telkom University, Indonesia. The Interior Design II course requires students to design the interior of a residence with a minimum area of 100 m2. The method of paired sample test analysis was used to assess student’s preferences for pre-test and post-test statements from the VR intervention method in assessing student’s sense of place in the final design of the course. The results showed significant differences in student preferences during the pre-test (481.3% and 790.6%), which increased during the post-test (641.7% and 801%). The paired sample t-test analysis results also showed a Sig (2-tailed) number of 0.000 < 0.05, so there is a significant relationship between the pre-test and post-test intervention.

The Second Life Metaverse and Its Usefulness in Medical Education After a Quarter of a Century.

The immersive virtual world platform Second Life (SL) was conceived 25 years ago, when Philip Rosedale founded Linden Lab in 1999 with the intention of developing computing hardware that would allow people to immerse themselves in a virtual world. This initial effort was transformed 4 years later into SL, a universally accessible virtual world centered on the user, with commercial transactions and even its own virtual currency, which fully connects with the concept of the metaverse, recently repopularized after the statements of the chief executive officer of Meta (formerly Facebook) in October 2021. SL is considered the best known virtual environment among higher education professionals. This paper aimed to review medical education in the SL metaverse; its evolution; and its possibilities, limitations, and future perspectives, focusing especially on medical education experiences during undergraduate, residency, and continuing medical education. The concept of the metaverse and virtual worlds was described, making special reference to SL and its conceptual philosophy, historical evolution, and technical aspects and capabilities for higher education. A narrative review of the existing literature was performed, including at the same time a point of view from our teaching team after an uninterrupted practical experience of undergraduate and postgraduate medical education in the last 13 years with >4000 users and >10 publications on the subject. From an educational point of view, SL has the advantages of being available 24/7 and creating in the student the important feeling of "being there" and of copresence. This, together with the reproduction of the 3D world, real-time interaction, and the quality of voice communication, makes the immersive experiences unique, generating engagement and a fluid interrelation of students with each other and with their teachers. Various groups of researchers in medical education have developed experiences during these years, which have shown that courses, seminars, workshops and conferences, problem-based learning experiences, evaluations, teamwork, gamification, medical simulation, and virtual objective structured clinical examinations can be successfully carried out. Acceptance from students and faculty is generally positive, recognizing its usefulness for undergraduate medical education and continuing medical education. In the 25 years since its conception, SL has proven to be a virtual platform that connects with the concept of the metaverse, an interconnected, open, and globally accessible system that all humans can access to socialize or share products for free or using a virtual currency. SL remains active and technologically improved since its creation. It is necessary to continue carrying out educational experiences, outlining the organization, objectives, and content and measuring the actual educational impact to make SL a tool of more universal use.

CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets

In the realm of digital creativity, our potential to craft intricate 3D worlds from imagination is often hampered by the limitations of existing digital tools, which demand extensive expertise and efforts. To narrow this disparity, we introduce CLAY, a 3D geometry and material generator designed to effortlessly transform human imagination into intricate 3D digital structures. CLAY supports classic text or image inputs as well as 3D-aware controls from diverse primitives (multi-view images, voxels, bounding boxes, point clouds, implicit representations, etc). At its core is a large-scale generative model composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a diverse range of 3D geometries. Specifically, it adopts neural fields to represent continuous and complete surfaces and uses a geometry generative module with pure transformer blocks in latent space. We present a progressive training scheme to train CLAY on an ultra large 3D model dataset obtained through a carefully designed processing pipeline, resulting in a 3D native geometry generator with 1.5 billion parameters. For appearance generation, CLAY sets out to produce physically-based rendering (PBR) textures by employing a multi-view material diffusion model that can generate 2K resolution textures with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for a range of controllable 3D asset creations, from sketchy conceptual designs to production ready assets with intricate details. Even first time users can easily use CLAY to bring their vivid 3D imaginations to life, unleashing unlimited creativity.

Spatial resolution measurement method for 3D displays from contrast modulation

Augmented Reality 3D head-up displays use a autostereoscopic 3D display as a panel. The 3D optical unit of autostereoscopic 3D displays controls the direction of the light rays in each pixel, allowing the users enjoy 3D world without glasses. However, these 3D optics cause image quality degradation. Deterioration of resolution has a serious impact on 3D image quality. Therefore, it is important to properly measure the 3D resolution according to 3D optics and analyze its impact. In this study, a method for measuring spatial resolution in 3D displays using contrast modulation is proposed. We describe a conventional 2D resolution measurement methods that are standardized. Based on the existing 2D resolution methods, we propose a 3D resolution method. The spatial and frequency signal responses of 3D displays were investigated. The first method is determined by the predominant frequency series. The second method is conducted by contrast modulation. Through experiments with 3D displays, 3D resolution was measured using the proposed method, and the relationship between the parameters and resolution of 3D optics was examined.

Automatyczna generacja wirtualnych elementów infrastruktury kolejowej

This article is devoted to the issues of modeling a 3D virtual world for railroad simulators and the problems of creating a map for such a simulator. An algorithm using artificial intelligence methods for detection, classification, and place railway infrastructure objects from video recordings in a 3D virtual world, as well as GPS data has been proposed. The proposed solution, supporting automatic generation of virtual elements of railway infrastructure, is a significant innovation in the field of research.

Parameter-Efficient Person Re-Identification in the 3D Space.

People live in a 3D world. However, existing works on person re-identification (re-id) mostly consider the semantic representation learning in a 2D space, intrinsically limiting the understanding of people. In this work, we address this limitation by exploring the prior knowledge of the 3D body structure. Specifically, we project 2D images to a 3D space and introduce a novel parameter-efficient omni-scale graph network (OG-Net) to learn the pedestrian representation directly from 3D point clouds. OG-Net effectively exploits the local information provided by sparse 3D points and takes advantage of the structure and appearance information in a coherent manner. With the help of 3D geometry information, we can learn a new type of deep re-id feature free from noisy variants, such as scale and viewpoint. To our knowledge, we are among the first attempts to conduct person re-id in the 3D space. We demonstrate through extensive experiments that the proposed method: (1) eases the matching difficulty in the traditional 2D space; 2) exploits the complementary information of 2D appearance and 3D structure; 3) achieves competitive results with limited parameters on four large-scale person re-id datasets; and 4) has good scalability to unseen datasets. Our code, models, and generated 3D human data are publicly available at https://github.com/layumi/person-reid-3d.

Learning layout generation for virtual worlds

The emergence of the metaverse has led to the rapidly increasing demand for the generation of extensive 3D worlds. We consider that an engaging world is built upon a rational layout of multiple land-use areas (e.g., forest, meadow, and farmland). To this end, we propose a generative model of land-use distribution that learns from geographic data. The model is based on a transformer architecture that generates a 2D map of the land-use layout, which can be conditioned on spatial and semantic controls, depending on whether either one or both are provided. This model enables diverse layout generation with user control and layout expansion by extending borders with partial inputs. To generate high-quality and satisfactory layouts, we devise a geometric objective function that supervises the model to perceive layout shapes and regularize generations using geometric priors. Additionally, we devise a planning objective function that supervises the model to perceive progressive composition demands and suppress generations deviating from controls. To evaluate the spatial distribution of the generations, we train an autoencoder to embed land-use layouts into vectors to enable comparison between the real and generated data using the Wasserstein metric, which is inspired by the Fréchet inception distance.

RIGA: Rotation-Invariant and Globally-Aware Descriptors for Point Cloud Registration.

Successful point cloud registration relies on accurate correspondences established upon powerful descriptors. However, existing neural descriptors either leverage a rotation-variant backbone whose performance declines under large rotations, or encode local geometry that is less distinctive. To address this issue, we introduce RIGA to learn descriptors that are Rotation-Invariant by design and Globally-Aware. From the Point Pair Features (PPFs) of sparse local regions, rotation-invariant local geometry is encoded into geometric descriptors. Global awareness of 3D structures and geometric context is subsequently incorporated, both in a rotation-invariant fashion. More specifically, 3D structures of the whole frame are first represented by our global PPF signatures, from which structural descriptors are learned to help geometric descriptors sense the 3D world beyond local regions. Geometric context from the whole scene is then globally aggregated into descriptors. Finally, the description of sparse regions is interpolated to dense point descriptors, from which correspondences are extracted for registration. To validate our approach, we conduct extensive experiments on both object- and scene-level data. With large rotations, RIGA surpasses the state-of-the-art methods by a margin of 8 ° in terms of the Relative Rotation Error on ModelNet40 and improves the Feature Matching Recall by at least 5 percentage points on 3DLoMatch.

Machine Learning in Metaverse Security: Current Solutions and Future Challenges

The Metaverse, positioned as the next frontier of the Internet, has the ambition to forge a virtual shared realm characterized by immersion, hyper-spatiotemporal dynamics, and self-sustainability. Recent technological strides in AI, Extended Reality, 6G, and blockchain propel the Metaverse closer to realization, gradually transforming it from science fiction into an imminent reality. Nevertheless, the extensive deployment of the Metaverse faces substantial obstacles, primarily stemming from its potential to infringe on privacy and be susceptible to security breaches, whether inherent in its underlying technologies or arising from the evolving digital landscape. Metaverse security provisioning is poised to confront various foundational challenges owing to its distinctive attributes, encompassing immersive realism, hyper-spatiotemporally, sustainability, and heterogeneity. This article undertakes a comprehensive study of the security and privacy challenges facing the Metaverse, leveraging machine learning models for this purpose. In particular, our focus centers on an innovative distributed Metaverse architecture characterized by interactions across 3D worlds. Subsequently, we conduct a thorough review of the existing cutting-edge measures designed for Metaverse systems while also delving into the discourse surrounding security and privacy threats. As we contemplate the future of Metaverse systems, we outline directions for open research pursuits in this evolving landscape.

PTVR - A software in Python to make virtual reality experiments easier to build and more reproducible.

Researchers increasingly use virtual reality (VR) to perform behavioral experiments, especially in vision science. These experiments are usually programmed directly in so-called game engines that are extremely powerful. However, this process is tricky and time-consuming as it requires solid knowledge of game engines. Consequently, the anticipated prohibitive effort discourages many researchers who want to engage in VR. This paper introduces the Perception Toolbox for Virtual Reality (PTVR) library, allowing visual perception studies in VR to be created using high-level Python script programming. A crucial consequence of using a script is that an experiment can be described by a single, easy-to-read piece of code, thus improving VR studies' transparency, reproducibility, and reusability. We built our library upon a seminal open-source library released in 2018 that we have considerably developed since then. This paper aims to provide a comprehensive overview of the PTVR software for the first time. We introduce the main objects and features of PTVR and some general concepts related to the three-dimensional (3D) world. This new library should dramatically reduce the difficulty of programming experiments in VR and elicit a whole new set of visual perception studies with high ecological validity.

Detection of maize stem diameter by using RGB-D cameras' depth information under selected field condition.

Stem diameter is a critical phenotypic parameter for maize, integral to yield prediction and lodging resistance assessment. Traditionally, the quantification of this parameter through manual measurement has been the norm, notwithstanding its tedious and laborious nature. To address these challenges, this study introduces a non-invasive field-based system utilizing depth information from RGB-D cameras to measure maize stem diameter. This technology offers a practical solution for conducting rapid and non-destructive phenotyping. Firstly, RGB images, depth images, and 3D point clouds of maize stems were captured using an RGB-D camera, and precise alignment between the RGB and depth images was achieved. Subsequently, the contours of maize stems were delineated using 2D image processing techniques, followed by the extraction of the stem's skeletal structure employing a thinning-based skeletonization algorithm. Furthermore, within the areas of interest on the maize stems, horizontal lines were constructed using points on the skeletal structure, resulting in 2D pixel coordinates at the intersections of these horizontal lines with the maize stem contours. Subsequently, a back-projection transformation from 2D pixel coordinates to 3D world coordinates was achieved by combining the depth data with the camera's intrinsic parameters. The 3D world coordinates were then precisely mapped onto the 3D point cloud using rigid transformation techniques. Finally, the maize stem diameter was sensed and determined by calculating the Euclidean distance between pairs of 3D world coordinate points. The method demonstrated a Mean Absolute Percentage Error (MAPE) of 3.01%, a Mean Absolute Error (MAE) of 0.75mm, a Root Mean Square Error (RMSE) of 1.07mm, and a coefficient of determination (R²) of 0.96, ensuring accurate measurement of maize stem diameter. This research not only provides a new method of precise and efficient crop phenotypic analysis but also offers theoretical knowledge for the advancement of precision agriculture.

Perceived User Reachability in Mobile UIs Using Data Analytics and Machine Learning

One-handed interactions on smartphone interfaces offer a prominent feature of highly mobile inputs. Thus, the design factor of user reachability is essential to realizing the incentive. However, the sole consideration of physical characteristics, such as hand size and thumb length, does not fully reflect the users’ perceived choices of hand poses and the corresponding inertia. We first conducted a 6-week questionnaire-based study of UI rating tasks and collected 62,156 responses reflecting user preferences for 3000 clustered UIs. Our analysis of the responses shows that user perceptions of smartphone UI components are divergent from their physical ability of thumb reaches; e.g. they can reach an icon with a thumb reach, but they prefer alternative hand poses. Accordingly, we propose a machine learning model, i.e. XGBoost (XGB), to predict the user’s choices of hand poses, with a reasonable prediction accuracy of 64% that can be regarded as a practical preliminary evaluation tool. With illustrative examples, our model can offer auxiliary information in the assessment of perceived user reachability with one-handed interaction on smartphone interfaces, which paves a path toward a computational understanding of UI designs, and such findings can be further extended to 2D UIs in 3D worlds.

Reliable prediction of difficult airway for tracheal intubation from patient preoperative photographs by machine learning methods

BackgroundEstimating the risk of a difficult tracheal intubation should help clinicians in better anaesthesia planning, to maximize patient safety. Routine bedside screenings suffer from low sensitivity. ObjectiveTo develop and evaluate machine learning (ML) and deep learning (DL) algorithms for the reliable prediction of intubation risk, using information about airway morphology. MethodsObservational, prospective cohort study enrolling n=623 patients who underwent tracheal intubation: 53/623 difficult cases (prevalence 8.51%).First, we used our previously validated deep convolutional neural network (DCNN) to extract 2D image coordinates for 27 + 13 relevant anatomical landmarks in two preoperative photos (frontal and lateral views). Here we propose a method to determine the 3D pose of the camera with respect to the patient and to obtain the 3D world coordinates of these landmarks. Then we compute a novel set of dM=59 morphological features (distances, areas, angles and ratios), engineered with our anaesthesiologists to characterize each individual's airway anatomy towards prediction.Subsequently, here we propose four ad hoc ML pipelines for difficult intubation prognosis, each with four stages: feature scaling, imputation, resampling for imbalanced learning, and binary classification (Logistic Regression, Support Vector Machines, Random Forests and eXtreme Gradient Boosting). These compound ML pipelines were fed with the dM=59 morphological features, alongside dD=7 demographic variables. Here we trained them with automatic hyperparameter tuning (Bayesian search) and probability calibration (Platt scaling). In addition, we developed an ad hoc multi-input DCNN to estimate the intubation risk directly from each pair of photographs, i.e. without any intermediate morphological description.Performance was evaluated using optimal Bayesian decision theory. It was compared against experts' judgement and against state-of-the-art methods (three clinical formulae, four ML, four DL models). ResultsOur four ad hoc ML pipelines with engineered morphological features achieved similar discrimination capabilities: median AUCs between 0.746 and 0.766. They significantly outperformed both expert judgement and all state-of-the-art methods (highest AUC at 0.716). Conversely, our multi-input DCNN yielded low performance due to overfitting. This same behaviour occurred for the state-of-the-art DL algorithms. Overall, the best method was our XGB pipeline, with the fewest false negatives at the optimal Bayesian decision threshold. ConclusionsWe proposed and validated ML models to assist clinicians in anaesthesia planning, providing a reliable calibrated estimate of airway intubation risk, which outperformed expert assessments and state-of-the-art methods. Our novel set of engineered features succeeded in providing informative descriptions for prognosis.

Neural Radiance Fields (NeRF) for 3D Visualization Rendering Based on 2D Images

We are living in a 3D world, but oftentimes we visualize things in 2D scopes like in pictures and videos of different objects. The world of 3D visualization has been constantly evolving over the past few years, still growing with the introduction of cutting-edge computer vision and deep learning methods. Reconstructing 3D scenes and being able to interact with them in virtual environments used to be something beyond our capabilities but now with the use of Neural Radiance Fields (NeRF) and Instant Neural Graphic Primitives (InstantNGP) we are able to create a world in a virtual space, opening the gateways to applications in object detection and generation, augmented reality, medical imaging, and more. Access to these methods will allow for more immersive development by creators, hands-on educational resources, and more experimentation and research, stimulating future applications and techniques.

Shape configuration of mental target representation as a holistic measure in a 3D real world pointing test for spatial orientation

Shape configuration of mental target representation as a holistic measure in a 3D real world pointing test for spatial orientation

Virtual Reality is Beneficial in Decreasing Pain in Laboring Women: A Preliminary Study

(Aust N Z J Obstet Gynaecol. 2023; 63: 193–197) Labor pain is a highly variable experience, with a wide range of both pharmacological and nonpharmacologic methods available to help relieve pain during labor. Virtual reality (VR) distraction therapy is one of many nonpharmacologic modalities shown to assist patients in managing their acute pain in clinical settings. This has been tested in patients undergoing chemotherapy, physical therapy, treatment for dental work, and burn wounds, among other procedures. Those who use VR technology wear a headset to experience a 3D multisensory virtual world. Four published quantitative studies to date have used VR to determine the influence of VR on laboring women’s pain scores. This preliminary study is an open-label, randomized, controlled cross-over trial to examine how effective VR is in decreasing pain in laboring women.