Harnessing the Metaverse in Modern Medicine: Virtual, Augmented, and Extended Reality as Catalysts for Healthcare Innovation and Education

Biopharma Is Going Digital … Bit by Bit

Objective/Scope Simplify and semi-automate the creation of Intelligent Digital Twins available on an augmented hybrid solution (on/offline) based on the latest mobile solutions and wearable technologies (Smart Glasses) communicating over Bluetooth. The main objective was to implement an interactive environment of digital twins through Augmented Reality (AR) to improve decision-making, enhance performance and optimize efficiencies in Inspection and Maintenance. Methods, Procedures, Process A breakthrough algorithm enabled to semi-automate the creation of 3D models by providing a platform of Intelligent Digital Twins to connect to various disparate systems and use Artificial Intelligence-driven augmentation techniques to contextualize and enrich the model with Tag numbers from the asset register. An application has been developed for Smart Glasses by transforming the Digital Twins into sectioned files to enable a performant AR experience and an on/offline syncing functionality with the mobile solution over Bluetooth. Furthermore, the 3D model was embedded within the mobile application for visual support during Inspection and Maintenance. Results, Observations, Conclusions As a result, the following products have been developed to improve efficiency and productivity by leveraging innovative technologies: Application for Intelligent Digital Twins: Single Source of Truth for managing and maintaining Intelligent Digital Twins using breakthrough technology to semi-automate the creation of 3D models from Point Cloud data. A Mark-Up Tool and a Contextualization Editor were developed on the promise of Digital Twins to introduce a communication, collaboration, and planning tool. Furthermore, the 3D model has been embedded within the mobile application environment for visual support on the field tablets. Application for Smart Glasses (AR Headset): An application has been developed for the latest AR Headset to enable an Augmented Reality experience and a Mark-Up tool with Bluetooth communication to the mobile application as well as to support a hybrid on/offline syncing functionality as well as data capturing on the fly. With its basic functionality of hands-free video streaming and communication functionality, the AR Headset enabled a Remote Pre-Commissioning, Certification, and Audit with vendors and independent certifiers during the pandemic. Application Programming Interfaces (API): API interfaces were established to various source systems like the Computerized Maintenance Management System, Inspection Data Management System, Drawing System, and the Process Information System to integrate and visualize the big data within the platform of Intelligent Digital Twins. Novel/Additive Information The use of automation and artificial intelligence (AI) driven technologies for creating Intelligent Digital Twins is a key enabler for global scalability and accelerates creating the company's digital backbone during the digital transformation. Enhancing this technology with an Augmented Reality experience, including mark-up annotations, provides a solid basis for data insights, data-driven decision making, and performance optimization.

BACKGROUND A digital twin (DT) can be understood as a representation of a real asset, in other words, a virtual replica of a physical object, process or even a system. Virtual models, can integrate with all the latest technologies, such as the Internet of Things (IoT), cloud computing and artificial intelligence (AI). Digital twins have applications in various sectors, ranging from manufacturing and engineering to healthcare. They have been used in managing healthcare facilities, streamlining care processes, personalizing treatments, and enhancing patient recovery. By analysing data from sensors and other sources, healthcare professionals can develop virtual models of patients, organs, and human systems, experimenting with various strategies to identify the most effective approach. This approach can lead to more targeted and efficient therapies while reducing the risk of collateral effects . Digital twin technology can also be used to generate a virtual replica of a hospital to review operational strategies, capabilities, personnel, and care models to identify areas for improvement, predict future challenges, and optimize organizational strategies. The potential impact of this tool on our society and its well-being is quite significant. OBJECTIVE The objective of the article is to present a general overview of the use of digital twins in healthcare METHODS With the aim of analysing and investigating the use of digital twins in health, a quick literature review was conducted on the topic in question using the following mesh terms: “digital twins”, “digital health”, and “health care”. Another filter applied in the search strategy was publications within a time range of up to 5 years (2018 to 2023). The search was conducted in six academic databases: IEEE Xplore, Dimensions, Scopus, Web of Science, PubMed and ACM. After applying the search strings and the exclusion criteria, a total of 58 publications were identified. The exclusion criteria described in the article were applied, resulting in 13 publications listed to constitute and support the discussion of this article. RESULTS The selected studies can be categorized according to the application of digital twins in the health sector into 2 groups: the clinical applications group, with 7 records, and the operational applications group, with 6 records. In the clinical applications group, five articles focused on the theme of personalized care/precision medicine, one related to the reproduction of biological structures and one focused on ethics issues related to the use of DTs in healthcare. In the operational applications group, we have a subgroup, with five articles that discuss the application of digital twins supporting the optimization of operational processes and another subgroup with one article that relies on the construction of virtual structures such as a hospital. CONCLUSIONS The use of digital twins, in process optimization and healthcare, presents important challenges related to data integration, privacy and interoperability. However, trends indicate exciting potential in personalizing treatment, prevention, remote monitoring, informed decision-making, and process management, which can result in significant improvements in quality and efficiency in healthcare. This work could, in some way, contribute to expanding discussions on the topic, opening space for new reflections. More in-depth future studies should be carried out to explore the possible consolidation of the use of digital twins in health, especially in processes linked to health care and primary health care, or even clarify which initiatives should be implemented or even strengthened to sustain the progress achieved thus far. CLINICALTRIAL not applicable

Digital twins, innovative virtual models synthesizing real-time biological, environmental, and lifestyle data, herald a new era in personalized medicine, particularly dermatology. These models, integrating medical-purpose Internet of Things (IoT) devices, deep and digital phenotyping, and advanced artificial intelligence (AI), offer unprecedented precision in simulating real-world physical conditions and health outcomes. Originating in aerospace and manufacturing for system behavior prediction, their application in healthcare signifies a paradigm shift towards patient-specific care pathways. In dermatology, digital twins promise enhanced diagnostic accuracy, optimized treatment plans, and improved patient monitoring by accommodating the unique complexities of skin conditions. However, a comprehensive review across PubMed, Embase, Web of Science, Cochrane, and Scopus until February 5th, 2024, underscores a significant research gap; no direct studies on digital twins’ application in dermatology is identified. This gap signals challenges, including the intricate nature of skin diseases, ethical and privacy concerns, and the necessity for specialized algorithms. Overcoming these barriers through interdisciplinary efforts and focused research is essential for realizing digital twins’ potential in dermatology. This study advocates for a proactive exploration of digital twins, emphasizing the need for a tailored approach to dermatological care that is as personalized as the patients themselves.

Digital twin is an efficient technology. With digital twin evolving, various sectors can gain advantages from it. The advancement of the Internet of Things (IoT), has helped digital twin to evolve. A digital twin is identical in all aspects with its original counterpart and can be used in running simulation, predicting errors of objects, products, etc. In connection to the healthcare sector, digital twin can help in it in different ways. Just like how it can provide advantages to various sectors, namely manufacturing sector, aviation sector, etc., similarly it can provide advantages to the healthcare sector as well. Using this technology, the healthcare sector can benefit largely. In this chapter, a literature based survey is presented on the application of Digital Twin and its challenges in healthcare. It is found that there are different applications of digital twin in healthcare sector. A real life example of the use of digital twin in healthcare application is also discussed and explained briefly in this chapter. The challenges digital twin face in the healthcare are also not just briefly discussed but also explained in this chapter.

A Digital Twin (DT) can be understood as a representation of a real asset, a virtual replica of a physical object, process, or even a system. They have been used in managing healthcare facilities, streamlining care processes, personalizing treatments, and enhancing patient recovery. The potential impact of this tool on our society and its well-being is quite significant. A quick review of the literature was carried out using the terms ('Digital Twins') and ('Digital Health'), and (Health Care) with a time interval of up to 5 years (2018-23). Using the PRISMA Method, the search was conducted in six academic databases: IEEE Xplore, Dimensions, Scopus, Web of Science, PubMed, and ACM. After applying the search strings and the exclusion criteria, a total of 13 publications were identified and listed to constitute and support the discussion of this article. The selected studies were categorized into 2 groups according to their application in healthcare: A group of clinical applications, subdivided into topics on personalized care and reproduction of biological structures and another group of operational applications, subdivided into topics such as optimization of operational processes, reproduction of physical structures, and development of devices and drugs. The use of DT in healthcare presents important challenges related to data integration, privacy, and interoperability. However, trends indicate exciting potential in personalizing treatment, prevention, remote monitoring, informed decision-making, and process management, which can result in significant improvements in quality and efficiency in healthcare.

<p dir="ltr">The deadly coronavirus disease (COVID-19) has highlighted the importance of remote health monitoring (RHM). The digital-twins (DTs) paradigm enables RHM by creating a virtual replica that receives data from the physical asset, representing its real-world behavior. However, DTs use passive Internet of Things (IoT) sensors, which limit their potential to a specific location or entity. This problem can be addressed by using the Internet of Robotic Things (IoRT), which combines robotics and IoT, allowing the robotic things (RTs) to navigate in a particular environment and connect to IoT devices in the vicinity. Implementing DTs in IoRT, creates a virtual replica [virtual twin (VT)] that receives real-time data from the physical RT [physical twin (PT)] to mirror its status. However, DTs require a user interface for real-time interaction and visualization. Virtual reality (VR) can be used as an interface due to its natural ability to visualize and interact with DTs. This research proposes a real-time system for RHM of COVID-19 patients using the DTs-based IoRT and VR-based user interface. It also presents and evaluates robot navigation performance, which is vital for remote monitoring. The VT operates the PT in the real environment (RE), which collects data from the patient-mounted sensors and transmits it to the control service to visualize in VR for medical examination. The system prevents direct interaction of medical staff with contaminated patients, protecting them from infection and stress. The experimental results verify the monitoring data quality (accuracy, completeness, and timeliness) and high accuracy of PT’s navigation.</p><h2>Other Information</h2><p dir="ltr">Published in: IEEE Internet of Things Journal<br>License: <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank">https://creativecommons.org/licenses/by/4.0/</a><br>See article on publisher's website: <a href="https://dx.doi.org/10.1109/jiot.2023.3267171" target="_blank">https://dx.doi.org/10.1109/jiot.2023.3267171</a></p>

The health care industry must align with new digital technologies to respond to existing and new challenges. Digital twins (DTs) are an emerging technology for digital transformation and applied intelligence that is rapidly attracting attention. DTs are virtual representations of products, systems, or processes that interact bidirectionally in real time with their actual counterparts. Although DTs have diverse applications from personalized care to treatment optimization, misconceptions persist regarding their definition and the extent of their implementation within health systems. This study aimed to review DT applications in health care, particularly for clinical decision-making (CDM) and operational decision-making (ODM). It provides a definition and framework for DTs by exploring their unique elements and characteristics. Then, it assesses the current advances and extent of DT applications to support CDM and ODM using the defined DT characteristics. We conducted a scoping review following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) protocol. We searched multiple databases, including PubMed, MEDLINE, and Scopus, for original research articles describing DT technologies applied to CDM and ODM in health systems. Papers proposing only ideas or frameworks or describing DT capabilities without experimental data were excluded. We collated several available types of information, for example, DT characteristics, the environment that DTs were tested within, and the main underlying method, and used descriptive statistics to analyze the synthesized data. Out of 5537 relevant papers, 1.55% (86/5537) met the predefined inclusion criteria, all published after 2017. The majority focused on CDM (75/86, 87%). Mathematical modeling (24/86, 28%) and simulation techniques (17/86, 20%) were the most frequently used methods. Using International Classification of Diseases, 10th Revision coding, we identified 3 key areas of DT applications as follows: factors influencing diseases of the circulatory system (14/86, 16%); health status and contact with health services (12/86, 14%); and endocrine, nutritional, and metabolic diseases (10/86, 12%). Only 16 (19%) of 86 studies tested the developed system in a real environment, while the remainder were evaluated in simulated settings. Assessing the studies against defined DT characteristics reveals that the developed systems have yet to materialize the full capabilities of DTs. This study provides a comprehensive review of DT applications in health care, focusing on CDM and ODM. A key contribution is the development of a framework that defines important elements and characteristics of DTs in the context of related literature. The DT applications studied in this paper reveal encouraging results that allow us to envision that, in the near future, they will play an important role not only in the diagnosis and prevention of diseases but also in other areas, such as efficient clinical trial design, as well as personalized and optimized treatments.

Digital twin applications in healthcare for people living with disability.

A comprehensive overview of developments in augmented reality, virtual reality, and mixed reality—and how they could affect every part of our lives. After years of hype, extended reality—augmented reality (AR), virtual reality (VR), and mixed reality (MR)—has entered the mainstream. Commercially available, relatively inexpensive VR headsets transport wearers to other realities—fantasy worlds, faraway countries, sporting events—in ways that even the most ultra-high-definition screen cannot. AR glasses receive data in visual and auditory forms that are more useful than any laptop or smartphone can deliver. Immersive MR environments blend physical and virtual reality to create a new reality. In this volume in the MIT Press Essential Knowledge series, technology writer Samuel Greengard offers an accessible overview of developments in extended reality, explaining the technology, considering the social and psychological ramifications, and discussing possible future directions. Greengard describes the history and technological development of augmented and virtual realities, including the latest research in the field, and surveys the various shapes and forms of VR, AR, and MR, including head-mounted displays, mobile systems, and goggles. He examines the way these technologies are shaping and reshaping some professions and industries, and explores how extended reality affects psychology, morality, law, and social constructs. It's not a question of whether extended reality will become a standard part of our world, he argues, but how, when, and where these technologies will take hold. Will extended reality help create a better world? Will it benefit society as a whole? Or will it merely provide financial windfalls for a select few? Greengard's account equips us to ask the right questions about a transformative technology.

The rapid development of digitalization, the Internet of Things (IoT), and Industry 4.0 has led to the emergence of the digital twin concept. IoT is an important pillar of the digital twin. The digital twin serves as a crucial link, merging the physical and digital territories of Industry 4.0. Digital twins are beneficial to numerous industries, providing the capability to perform advanced analytics, create detailed simulations, and facilitate informed decision-making that IoT supports. This paper presents a review of the literature on digital twins, discussing its concepts, definitions, frameworks, application methods, and challenges. The review spans various domains, including manufacturing, energy, agriculture, maintenance, construction, transportation, and smart cities in Industry 4.0. The present study suggests that the terminology “3 dimensional (3D) digital twin” is a more fitting descriptor for digital twin technology assisted by IoT. The aforementioned statement serves as the central argument of the study. This article advocates for a shift in terminology, replacing “digital twin” with “3D digital twin” to more accurately depict the technology’s innate potential and capabilities in Industry 4.0. We aim to establish that “3D digital twin” offers a more precise and holistic representation of the technology. By doing so, we underline the digital twin’s analytical ability and capacity to offer an intuitive understanding of systems, which can significantly streamline decision-making processes using the digital twin.

The accelerating complexity of urban environments has prompted cities to adopt digital technologies that improve efficiency, sustainability, and resilience. Among these, Urban Digital Twins (UDTw) have emerged as transformative tools for real-time representation, simulation, and management of urban systems. This Systematic Literature Review (SLR) examines the integration of Digital Twins (DTw), the Internet of Things (IoT), and Artificial Intelligence (AI) into the Smart City Development (SCD). Following the PSALSAR framework and PRISMA 2020 guidelines, 64 peer-reviewed articles from IEEE Xplore, Association for Computing Machinery (ACM), Scopus, and Web of Science (WoS) digital libraries were analyzed by using bibliometric and thematic methods via the Bibliometrix package in R. The review allowed identifying key technological trends, such as edge–cloud, architectures, 3D immersive visualization, Generative AI (GenAI), and blockchain, and classifies UDTw applications into five domains: traffic management, urban planning, environmental monitoring, energy systems, and public services. Persistent challenges have been also outlined, including semantic interoperability, predictive modeling, data privacy, and impact evaluation. This study synthesizes the current state of the field, by clearly identifying a thematic mapping, and proposes a research agenda to align technical innovation with measurable urban outcomes, offering strategic insights for researchers, policymakers, and planners.

Abstract. Digital Twins as virtual representations of industrial assets are being used to assimilate varied sources of data for improved awareness and decision making in operations and process optimisation. This paper explores the integration of IoT sensors into a spatial digital twin called Fuse that Woodside Energy has been building for the assets it operates. We describe the Fuse platform and its knowledge graph data core that is used to organise and inter-relate data for presentation within 3D visualisations, domain-specific contexts and immersive augmented reality presentations. The key contribution here is the use of a knowledge graph to link diverse data sources so as to contextualise sensor data for actionable insights. One area of significant innovation has been the development and use of new Internet of Things (IoT) devices which have been enabled by advances in sensor technology, connectivity, and cloud computing. These new tailored data sources are complimenting existing plant and resource planning data for improved asset monitoring, predictive maintenance and process automation.

In this chapter we look at Audio, a key aspect of virtual and augmented realities that, because of the dominance of the visual sense, is often overlooked in the creation of healthcare and other applications. Audio design and implementation is often under budgeted and considered at a late stage in many projects. This chapter seeks to raise awareness for the importance of high-quality audio in healthcare applications and describes the process of creating quality sound recordings. By way of an example, we describe research that uses nature sounds in conjunction with virtual environments for the purpose of managing stress, pain and other illnesses. We show how noise and distortions can have adverse effects on patient recovery and then describe how to avoid common mistakes in the recording of sounds. We discuss the equipment and workflow that is needed for the production of high-quality sound recordings for use in contemporary healthcare applications.

Chapter 10 - Impact of internet of things and digital twin on manufacturing era