Immersive Communication Research Articles

Tactile Sensing and Rendering Patch with Dynamic and Static Sensing and Haptic Feedback for Immersive Communication.

Wearable human-machine interface (HMI) with bidirectional and multimodal tactile information exchange is of paramount importance in teleoperation by providing more intuitive data interpretation and delivery of tactilely related signals. However, the current sensing and feedback devices still lack enough integration and modalities. Here, we present a Tactile Sensing and Rendering Patch (TSRP) that is made of a customized expandable array which consists of a piezoelectric sensing and feedback unit fused with an elastomeric triboelectric multidimensional sensor and its inner pneumatic feedback structure. The primary functional unit of TSRP is mainly featured with a soft silicone substrate with compact multilayer structure integrating static and dynamic multidimensional tactile sensing capabilities, which synergistically leverage both triboelectric and piezoelectric effects. Additionally, based on the air chamber created by the triboelectric sensor and the converse piezoelectric effect, it provides pneumatic and vibrational haptic feedback simultaneously for both static and dynamic perception regeneration. With the aid of the other variants of this unit, the array shaped TSRP is capable of simulating different terrains, geometries, sliding, collisions, and other critical interactive events during teleoperation via skin perception. Moreover, immediate manipulation can be done on TSRP through the tactile sensors. The preliminary demonstration of TSRP interface with a completed control module in robotic teleoperation is provided, which shows the feasibility of assisting certain tasks in a complex environment by direct tactile communication. The proposed device offers a potential method of enabling bidirectional tactile communication with enriched key information for improving interaction efficiency in the fields of robot teleoperation and training.

FROM STORYTELLING TO STORY LIVING: EXPLORING THE METAVERSE AND ITS IMPLICATIONS FOR THE NEW AGE COMMUNICATOR

In the unfolding landscape of the Fifth Industrial Revolution, the concept of the "Metaverse" has found resonance within our collective consciousness. This paper presents a critical examination of the transformative potential of the Metaverse in the context of media and technology, emphasizing its implications for the field of communications. Rooted in a rhizomatic paradigm marked by continuous transformation, the Metaverse encapsulates augmented and virtual reality, blockchain, artificial Intelligence, and Web3.0 technologies. This study focuses on the Metaverse through the lens of the communicator, exploring immersive experiences powered by technological convergence and their potential impact on content.Building upon Baker et al.'s (2019) foundational insights, the research investigates the Metaverse's influence on media engagement and societal shifts. Nicholson et al.'s (2014) findings on collaboration within virtual world technology and Petit et al.'s (2019) exploration of consumer acceptance of digital objects provide a nuanced understanding of immersive communication. In the post-pandemic era, marked by rapid digital acceleration, the paper evaluates the Metaverse's role in shaping user behavior and societal structures, shedding light on its profound impact on media consumption patterns.Delving into the convergence of media and technology within the Metaverse, the paper explores its potential implications on immersive storytelling. It scrutinizes the evolving landscape where reification, presence, and immersion converge to give rise to the emergent scenario of "Story Living," blurring the boundaries between reality and simulation.The Metaverse prompts a deep and dynamic engagement with audiences and places them at the heart of the content piece. The introduction of interactivity and adaptive storytelling, facilitated by mixed reality technologies, empowers audiences to actively shape and participate in the narrative. This shift underscores the importance of adaptive multi-varied formats of communication within the immersive context of the Metaverse. Shin & Biocca's (2017) research on virtual world, Chelsea et al.'s formulation of Hierarchy of Needs for User Experiences in Virtual Reality, as well as Thomas and David’s (1992) insights about modeling immersive environments provide a comprehensive analysis of the intersection between media, technology, and consumer experiences within this dynamic space. The paper seeks inspiration from Ahn et.al's framework on bifold, triadic relationships to understand the nuances of human interactions unfolding in the metaverse.The paper acknowledges the adoption of the Metaverse beyond traditional gaming and corporate training. The integration of immersive environments by fashion brands, and investments by major brands in this space as outlined by Rauschnabel et al., (2022) signal an ongoing shift in brand focus. Shen's (2002) examination of measurement methodologies and strategic considerations, underscores the need for critical assessment in academic discourse. By grasping the opportunities and challenges presented by the empowered audience within the Metaverse, this study enriches discussions on the cultural and societal impacts of the Metaverse for the new-age media communicator.

Impact of LiDAR point cloud compression on 3D object detection evaluated on the KITTI dataset

The rapid growth on the amount of generated 3D data, particularly in the form of Light Detection And Ranging (LiDAR) point clouds (PCs), poses very significant challenges in terms of data storage, transmission, and processing. Point cloud (PC) representation of 3D visual information has shown to be a very flexible format with many applications ranging from multimedia immersive communication to machine vision tasks in the robotics and autonomous driving domains. In this paper, we investigate the performance of four reference 3D object detection techniques, when the input PCs are compressed with varying levels of degradation. Compression is performed using two MPEG standard coders based on 2D projections and octree decomposition, as well as two coding methods based on Deep Learning (DL). For the DL coding methods, we used a Joint Photographic Experts Group (JPEG) reference PC coder, that we adapted to accept LiDAR PCs in both Cartesian and cylindrical coordinate systems. The detection performance of the four reference 3D object detection methods was evaluated using both pre-trained models and models specifically trained using degraded PCs reconstructed from compressed representations. It is shown that LiDAR PCs can be compressed down to 6 bits per point with no significant degradation on the object detection precision. Furthermore, employing specifically trained detection models improves the detection capabilities even at compression rates as low as 2 bits per point. These results show that LiDAR PCs can be coded to enable efficient storage and transmission, without significant object detection performance loss.

Diffusion of innovation in direct-to-avatar (D2A): A study of immersive communication and branding in the metaverse

Direct-to-Avatar (D2A) is the new phenomenon in the online retail segment. Brands are moving towards immersive commerce in which the central focus is on immersive communication for better customer experience in the metaverse. Diffusion of innovation theory acted as a theoretical framework to understand the premiums that the brands can enjoy from the customers by moving towards immersive commerce. 948 tweets on metaverse were analysed and five in-depth interviews with the experts on metaverse were conducted. For the quantitative study, the data were collected from a select metaverse Facebook group. The findings of PLS-SEM revealed that four factors - customer engagement, customer experience, value co-creation and customer empowerment - have statistically positive impact on customer citizenship behavior which ultimately affects customer satisfaction and loyalty towards the brands. The study extends the applications of diffusion of innovation theory in the content of immersive communication. It enriches the literature on technological advancements in immersive commerce. This is one of the first studies that focus on lifestyle brands in the metaverse. Also, this study depicts how socializing the digital brand communication with experience economy can be a key factor for lifestyle brands.

Scalable Multiuser Immersive Communications With Multi-Numerology and Mini-Slot

Scalable Multiuser Immersive Communications With Multi-Numerology and Mini-Slot

Underwater Gesture Recognition Meta-Gloves for Marine Immersive Communication.

Rapid advancements in immersive communications and artificial intelligence have created a pressing demand for high-performance tactile sensing gloves capable of delivering high sensitivity and a wide sensing range. Unfortunately, existing tactile sensing gloves fall short in terms of user comfort and are ill-suited for underwater applications. To address these limitations, we propose a flexible hand gesture recognition glove (GRG) that contains high-performance micropillar tactile sensors (MPTSs) inspired by the flexible tube foot of a starfish. The as-prepared flexible sensors offer a wide working range (5 Pa to 450 kPa), superfast response time (23 ms), reliable repeatability (∼10000 cycles), and a low limit of detection. Furthermore, these MPTSs are waterproof, which makes them well-suited for underwater applications. By integrating the high-performance MPTSs with a machine learning algorithm, the proposed GRG system achieves intelligent recognition of 16 hand gestures under water, which significantly extends real-time and effective communication capabilities for divers. The GRG system holds tremendous potential for a wide range of applications in the field of underwater communications.

Generative AI for Immersive Communication: The Next Frontier in Internet-of-Senses Through 6G

Generative AI for Immersive Communication: The Next Frontier in Internet-of-Senses Through 6G

The Intersection of Art and Technology: Innovative Thinking and Practice in the Perspective of New Media

Abstract The swift advancement of science and technology has emerged as a pivotal catalyst for artistic innovation, with immersive art displays becoming increasingly prevalent as a favored mode of creative expression. This paper conducts a thorough analysis of the flow dynamics within immersive art display spaces, utilizing the eight-channel cardio fluid flow model as a foundation. By integrating the immersive communication effect model, this study further investigates the communicative aspects of these spaces. The implementation of human-computer interaction technologies not only enhances visual presentation but also facilitates the design of immersive art spaces through advanced scene modeling techniques. This design process includes optimizing the viewer’s perspective by incorporating visual perception parameters and the grid saliency MS algorithm. Additionally, this research undertakes a quantitative evaluation of interactive applications at the nexus of art, science, and technology, focusing on audience demand, interactive experiences, and recognition. The findings reveal a significant increase in audience demand for digital media interaction, from 12.64% to 31.29%, and an impressive average tolerance score of 84.32 for immersive art spaces. Comments related to these spaces uniformly express positive recognition, underscoring the successful integration of science and technology in art displays. This integration, as demonstrated, fosters a diverse evolution of artistic expression forms.

An Analysis of Immersive Communication in VR Documentary: A Case Study of Planet Earth II

The rapid development of Virtual Reality (VR) technology has brought significant changes to the field of cultural media. Among them, VR documentaries, as an emerging form of communication, provide viewers with a brand-new sensory experience through their powerful immersive capabilities. This article will analyze the immersive communication features of VR documentaries, using the classic VR documentary Planet Earth II as an example, and explore its impact on viewers based on existing literature and data. By using VR headsets and 360-degree video capture technology, documentary filmmakers are able to immerse viewers in wildlife habitats and interact with the natural environment. This immersive experience enhances the audiences emotional empathy and engagement, giving them a deeper understanding and appreciation of the beauty and fragility of the natural world. In addition, as VR technology continues to evolve, it can be expected that future documentary productions will add elements of interactivity and personalization. Viewers may be able to choose what interests them, follow their favorite characters to explore, and interact with objects in the documentary. This personalized experience will make viewers more engaged and further drive innovation in documentary production.

CoMIC: A Collaborative Mobile Immersive Computing Infrastructure for Conducting Multi-User XR Research

While existing research on immersive technologies such as virtual, augmented, and mixed reality has been shifting from improving the quality of experience (QoE) of a single user to multi-user scenarios, which are naturally required by the burgeoning Metaverse, there is little work on building the underlying infrastructure to support multi-user immersive applications, especially for geo-distributed users. In this position paper, we propose a research infrastructure, dubbed CoMIC, to fill this critical gap, by offering a collaborative, visual-first, and hologram-based computing space and a QoE-driven, multi-site, and immersive communication framework. With its rudimentary prototype, we demonstrate the effectiveness of CoMIC through a case study of multi-user volumetric video streaming over mmWave. We conclude this paper with a discussion of numerous unique opportunities that can be enabled by CoMIC.

HDHumans

Photo-real digital human avatars are of enormous importance in graphics, as they enable immersive communication over the globe, improve gaming and entertainment experiences, and can be particularly beneficial for AR and VR settings. However, current avatar generation approaches either fall short in high-fidelity novel view synthesis, generalization to novel motions, reproduction of loose clothing, or they cannot render characters at the high resolution offered by modern displays. To this end, we propose HDHumans, which is the first method for HD human character synthesis that jointly produces an accurate and temporally coherent 3D deforming surface and highly photo-realistic images of arbitrary novel views and of motions not seen at training time. At the technical core, our method tightly integrates a classical deforming character template with neural radiance fields (NeRF). Our method is carefully designed to achieve a synergy between classical surface deformation and a NeRF. First, the template guides the NeRF, which allows synthesizing novel views of a highly dynamic and articulated character and even enables the synthesis of novel motions. Second, we also leverage the dense pointclouds resulting from the NeRF to further improve the deforming surface via 3D-to-3D supervision. We outperform the state of the art quantitatively and qualitatively in terms of synthesis quality and resolution, as well as the quality of 3D surface reconstruction.

Methodology to Assess Quality, Presence, Empathy, Attitude, and Attention in 360-degree Videos for Immersive Communications

This paper analyzes the joint assessment of quality, spatial and social presence, empathy, attitude, and attention in three conditions: (A)visualizing and rating the quality of contents in a Head-Mounted Display (HMD), (B)visualizing the contents in an HMD,and (C)visualizing the contents in an HMD where participants can see their hands and take notes. The experiment simulates an immersive communication where participants attend conversations of different genres and from different acquisition perspectives in the context of international experiences. Video quality is evaluated with Single-Stimulus Discrete Quality Evaluation (SSDQE) methodology. Spatial and social presence are evaluated with questionnaires adapted from the literature. Initial empathy is assessed with Interpersonal Reactivity Index(IRI) and a questionnaire is designed to evaluate attitude. Attention is evaluated with 3 questions that had pass/fail answers. 54 participants were evenly distributed among A, B, and C conditions taking into account their international experience backgrounds, obtaining a diverse sample of participants. The results from the subjective test validate the proposed methodology in VR communications, showing that video quality experiments can be adapted to conditions imposed by experiments focused on the evaluation of socioemotional features in terms of contents of long-duration, actor and observer acquisition perspectives, and genre. In addition, the positive results related to the sense of presence imply that technology can be relevant in the analyzed use case. The acquisition perspective greatly influences social presence and all the contents have a positive impact on all participants on their attitude towards international experiences. The annotated dataset, Student Experiences Around the World dataset (SEAW-dataset), obtained from the experiment is made publicly available.

Exploring Immersive Interpersonal Communication via AR

A central challenge of social computing research is to enable people to communicate expressively with each other remotely. Augmented reality has great promise for expressive communication since it enables communication beyond texts and photos and towards immersive experiences rendered in recipients' physical environments. Little research, however, has explored AR's potential for everyday interpersonal communication. In this work, we prototype an AR messaging system, ARwand, to understand people's behaviors and perceptions around communicating with friends via AR messaging. We present our findings under four themes observed from a user study with 24 participants, including the types of immersive messages people choose to send to each other, which factors contribute to a sense of immersiveness, and what concerns arise over this new form of messaging. We discuss important implications of our findings on the design of future immersive communication systems.

Two-Tier 360-Degree Video Delivery Control in Multiuser Immersive Communications Systems

In the immersive communications systems, video information is stringently delivered for extended reality applications. Since users may demand various immersive experience and have different levels of view prediction accuracy, the two-tier 360-degree video delivery frame structure with dynamic transmission time interval duration needs to be designed appropriately. To maximize the system's quality of experience, two new frame structures and power control are proposed. Meanwhile, the synchronous and asynchronous cases with different computational complexities are addressed. The results demonstrate the effectiveness of the proposed design and indicate that flexible frame structure is beneficial for multiuser immersive communications.

Toward immersive communications in 6G

The sixth generation (6G) networks are expected to enable immersive communications and bridge the physical and the virtual worlds. Integrating extended reality, holography, and haptics, immersive communications will revolutionize how people work, entertain, and communicate by enabling lifelike interactions. However, the unprecedented demand for data transmission rate and the stringent requirements on latency and reliability create challenges for 6G networks to support immersive communications. In this survey article, we present the prospect of immersive communications and investigate emerging solutions to the corresponding challenges for 6G. First, we introduce use cases of immersive communications, in the fields of entertainment, education, and healthcare. Second, we present the concepts of immersive communications, including extended reality, haptic communication, and holographic communication, their basic implementation procedures, and their requirements on networks in terms of transmission rate, latency, and reliability. Third, we summarize the potential solutions to addressing the challenges from the aspects of communication, computing, and networking. Finally, we discuss future research directions and conclude this study.

Synchronization of Speculative Speech and Gesture in Immersive Communication with Introverted and Extraverted Participants

Synchronization of Speculative Speech and Gesture in Immersive Communication with Introverted and Extraverted Participants

Learning-based Auction for Matching Demand and Supply of Holographic Digital Twin Over Immersive Communications

Learning-based Auction for Matching Demand and Supply of Holographic Digital Twin Over Immersive Communications

Robust triboelectric information‐mat enhanced by multi‐modality deep learning for smart home

AbstractIn metaverse, a digital‐twin smart home is a vital platform for immersive communication between the physical and virtual world. Triboelectric nanogenerators (TENGs) sensors contribute substantially to providing smart‐home monitoring. However, TENG deployment is hindered by its unstable output under environment changes. Herein, we develop a digital‐twin smart home using a robust all‐TENG based information mat (InfoMat), which consists of an in‐home mat array and an entry mat. The interdigital electrodes design allows environment‐insensitive ratiometric readout from the mat array to cancel the commonly experienced environmental variations. Arbitrary position sensing is also achieved because of the interval arrangement of the mat pixels. Concurrently, the two‐channel entry mat generates multi‐modality information to aid the 10‐user identification accuracy to increase from 93% to 99% compared to the one‐channel case. Furthermore, a digital‐twin smart home is visualized by real‐time projecting the information in smart home to virtual reality, including access authorization, position, walking trajectory, dynamic activities/sports, and so on.image

Soft Modular Glove with Multimodal Sensing and Augmented Haptic Feedback Enabled by Materials' Multifunctionalities.

Immersive communications rely on smart perception based on diversified and augmented sensing and feedback technologies. However, the increasing of functional components also raises the issue of increased system complexity. Here, we propose a modular soft glove with multimodal sensing and feedback functions by exploring and utilizing the multiple properties of glove materials. With a single design of basic structure, the main functional unit possesses triboelectric-based sensing of static and dynamic contact, vibration, strain, and pneumatic actuation. Additionally, the same unit is also capable of offering pneumatic tactile haptic feedback and electroresistive thermal haptic feedback. Together with a machine learning algorithm, the proposed glove not only performs real-time detection of dexterous hand motion and direct feedback but also realizes intelligent object recognition and augmented feedback, which significantly enhance the communication and perception of more comprehensive information. In general, this glove utilizes a facile designed sensing and feedback device to achieve dual-way and multimodal communication among humans, machines, and the virtual world via smart perceptions.

Emerging Immersive Communication Systems: Overview, Taxonomy, and Good Practices for QoE Assessment

Several technological and scientific advances have been achieved recently in the fields of immersive systems (e.g., 360-degree/multiview video systems, augmented/mixed/virtual reality systems, immersive audio-haptic systems, etc.), which are offering new possibilities to applications and services in different communication domains, such as entertainment, virtual conferencing, working meetings, social relations, healthcare, and industry. Users of these immersive technologies can explore and experience the stimuli in a more interactive and personalized way than previous technologies (e.g., 2D video). Thus, considering the new technological challenges related to these systems and the new perceptual dimensions and interaction behaviors involved, a deep understanding of the users’ Quality of Experience (QoE) is required to satisfy their demands and expectations. In this sense, it is essential to foster the research on evaluating the QoE of immersive communication systems, since this will provide useful outcomes to optimize them and to identify the factors that can deteriorate the user experience. With this aim, subjective tests are usually performed following standard methodologies (e.g., ITU recommendations), which are designed for specific technologies and services. Although numerous user studies have been already published, there are no recommendations or standards that define common testing methodologies to be applied to evaluate immersive communication systems, such as those developed for images and video. Taking this into account, a revision of the QoE evaluation methods designed for previous technologies is required to develop robust and reliable methodologies for immersive communication systems. Thus, the objective of this paper is to provide an overview of existing immersive communication systems and related user studies, which can help on the definition of basic guidelines and testing methodologies to be used when performing user tests of immersive communication systems, such as 360-degree video-based telepresence, avatar-based social VR, cooperative AR, etc.