Multisensory Technologies Research Articles

Theories and Methods for Indoor Positioning Systems: A Comparative Analysis, Challenges, and Prospective Measures

In the era of the Internet of Things (IoT), the demand for accurate positioning services has become increasingly critical, as location-based services (LBSs) depend on users’ location data to deliver contextual functionalities. While the Global Positioning System (GPS) is widely regarded as the standard for outdoor localization due to its reliability and comprehensive coverage, its effectiveness in indoor positioning systems (IPSs) is limited by the inherent complexity of indoor environments. This paper examines the various measurement techniques and technological solutions that address the unique challenges posed by indoor environments. We specifically focus on three key aspects: (i) a comparative analysis of the different wireless technologies proposed for IPSs based on various methodologies, (ii) the challenges of IPSs, and (iii) forward-looking strategies for future research. In particular, we provide an in-depth evaluation of current IPSs, assessing them through multidimensional matrices that capture diverse architectural and design considerations, as well as evaluation metrics established in the literature. We further examine the challenges that impede the widespread deployment of IPSs and highlight the potential risk that these systems may not be recognized with a single, universally accepted standard method, unlike GPS for outdoor localization, which serves as the golden standard for positioning. Moreover, we outline several promising approaches that could address the existing challenges of IPSs. These include the application of transfer learning, feature engineering, data fusion, multisensory technologies, hybrid techniques, and ensemble learning methods, all of which hold the potential to significantly enhance the accuracy and reliability of IPSs. By leveraging these advanced methodologies, we aim to improve the overall performance of IPSs, thus paving the way for more robust and dependable LBSs in indoor environments.

Learning from Users: Everyday Playful Interactions to Support Architectural Spatial Changes

While there's growing interest in eliciting situated playful interactions with technologies in different contexts, how these interactions might shape everyday spaces still needs to be fully explored. To address this gap, this article aims to guide the architectural spatial changes by exploring everyday playful interactions of technology-adopted users in domestic spaces. We present our contributions in a two-fold study: First, through an extensive diary study involving 13 technology-adopted residents in gated communities, where distinct boundaries offer increased opportunities for playful interactions, we identified four playful themes: (1) creating and expanding play-spaces, (2) balancing play and comfort, (3) intertwining imagination with spatial experience, and (4) gamifying household interactions. Secondly, by building on these themes, we outline three design implications to inform architectural design processes, aiming to translate everyday playful interactions into tangible spatial changes. These implications include adapting shape-changing and wearable technologies for playful flexibility, embedding new forms of communications within infrastructures, and turning homes into interactive entities with multi-sensory technologies. While these findings provide a starting point for exploring new architectural design possibilities in similar environments, further research with architects, policymakers, and design researchers in the Human-Computer Interaction (HCI) and Human-Building Interaction (HBI) fields is essential to actualizing these changes.

Multisensory Technologies for Inclusive Exhibition Spaces: Disability Access Meets Artistic and Curatorial Research

This article discusses applications of technology for sensory-disabled audiences in modern and contemporary art exhibitions. One case study of experimental artistic and curatorial research by The OtherAbilities art collective is discussed: a series of prototype tools for sensory translation from audible sound to vibration were developed to be embeddable in the architecture of spaces where art is presented. In the article, the case study is approached from a curatorial perspective. Based on bibliographical sources, the article starts with a brief historical reference to disability art activism and a presentation of contemporary accessibility solutions for sensory-disabled audiences in museums. The research for the case study was conducted during testing and feedback sessions on the prototypes using open-ended oral interviews, open-ended written comments, and ethnographic observation of visitors’ behavior during exhibitions. The testers were d/Deaf, hard of hearing and hearing. The results focus on the reception of the sensory translation of audible sound to vibration by test users of diverse hearing abilities and on the reception of the prototypes in the context of art and design exhibitions. The article closes with a reflection on how disability scholarship meets art curatorial theory in the example of the article’s case study.

The Impact of Sensory Marketing: Analysis of its Attributes Towards Online Perfum Users’ Behavior

Purpose: The tendency of consumers to purchase online today is enormous and requires the role of sensory marketing in providing a positive and memorable experience to all. In addition, since sensory marketing whether in online or offline settings can unconsciously govern shoppers’ judgement and purchase behavior, this paper aims to understand how multi-sensory information processing in the online environment is diverted from offline in the cosmetic perfume industry by leveraging previous research to analyze its effect on offline stores and emphasizing the present study on online retail stores. Theoretical framework: The research study conducted by Petit et al. (2019) attempts to explore more about digital sensory marketing and multisensory technologies. However, offline and online settings would have different effects and since perfume products require the use of our senses, there would be obstacles that prevent us from using our senses when sold online. This requires us to understand sensory marketing in the online shopping environment. Design/Methodology/Approach: This paper uses an exploratory study approach to gain in-depth understanding of the topic. The qualitative study is essentially built on primary data sources, through naturalistic observation and semi-structured interviews with 9 Generation Z individuals who are Indonesian citizens and have undertaken frequent perfume purchases online. A snowball sampling method was used in finding respondents for this study while interview results were analyzed using thematic analysis. Findings: The paper highlights the significant impact of sensory marketing through the use of sensorial cues, sensorial attributes, and individual differences in sensory perception. Ethical considerations when utilizing sensory marketing are also being emphasized throughout this study. Research, Practical & Social implications: Results of this study can encourage future research to help bring benefits for website/internet site designers, content managers, online perfume retailers, as well as academicians in general. Originality/Value: This paper draws attention to the growing use of online channels by cosmetic perfume industry actors and sheds light on the importance of sensory marketing practices in affecting online user behavior by answering what and how questions using exploratory studies.

Personal and ubiquitous computing, special issue: “Sonic experiences: interaction, connectivity, and multi-sensory technologies”

Personal and ubiquitous computing, special issue: “Sonic experiences: interaction, connectivity, and multi-sensory technologies”

Feeling Connected: The Role of Haptic Feedback in VR Concerts and the Impact of Haptic Music Players on the Music Listening Experience

Today, some of the most widely attended concerts are in virtual reality (VR). For example, the videogame Fortnite recently attracted 12.3 million viewers sitting in homes all over the world to a VR Travis Scott rap concert. As such VR concerts become increasingly ubiquitous, we are presented with an opportunity to design more immersive virtual experiences by augmenting VR with other multisensory technologies. Given that sound is a multi-modal phenomenon that can be experienced sonically and vibrationally, we investigated the importance of haptic feedback to musical experiences using a combination of qualitative and empirical methodologies. Study 1 was a qualitative study demonstrating that, unlike their live counterparts, current VR concerts make it harder for audiences to form a connection with artists and their music. Furthermore, VR concerts lack multisensory feedback and are perceived as less authentic than live concert experiences. Participants also identified a variety of different kinds of touch that they receive at live concerts and suggested that ideal VR concerts would replicate physical touch and thermal feedback from the audience, emotional touch, and vibrations from the music. Specifically, users advocated for the use of haptic devices to increase the immersiveness of VR concert experiences. Study 2 isolated the role of touch in the music listening experience and empirically investigated the impact of haptic music players (HMPs) on the audio-only listening experience. An empirical, between-subjects study was run with participants either receiving vibrotactile feedback via an HMP (haptics condition) or no vibrotactile feedback (control) while listening to music. Results indicated that listening to music while receiving vibrotactile feedback increased participants’ sense of empathy, parasocial bond, and loyalty towards the artist, while also decreasing participants’ feelings of loneliness. The connection between haptics condition and these dependent variables was mediated by the feeling of social presence. Study 2 thus provides initial evidence that HMPs may be used to meet people’s need for connection, multisensory immersion, and complex forms of touch in VR concerts as identified in Study 1.

Sensory disclosure in an augmented environment: memory of touch and willingness to buy

The recent movement restrictions due to the COVID-19 outbreak, and subsequent booming in online buying, show consumers the crucial role of touch as a pre-purchase indicator of good quality products, especially in grocery and fashion sectors. During the pandemic time, consumers were somehow forced to find different alternatives to understand the quality of items before buying. Among these, the adoption of multi-sensory technologies (e.g., augmented reality, AR) or the visual activation of the memory of a material (e.g., wool) was certainly attracting attention from consumers. In this scenario, the purpose of this paper is to explore how AR can activate the memory of products and consumers’ willingness to purchase. Building on qualitative data from semi-structured interviews with 18 Italian consumers, our analysis reveals that AR might enhance consumers’ memory of products, with this sometimes occurring unconsciously. Our findings further deepen the scientific and managerial debate around the role AR might have in influencing/altering consumers’ shopping routines. Contributions to the body of knowledge and managerial implications are discussed and suggestions for further research are given.

Efficacy of Multisensory Technology in Post-Stroke Cognitive Rehabilitation: A Systematic Review.

Post-stroke, in addition to sensorimotor signs and symptoms, could lead to cognitive deficits. Theories of embodiment stress the role of sensorimotor system and multisensory integration in sustaining high-order cognitive domains. Despite conventional post-stroke cognitive rehabilitation being effective, innovative technologies could overcome some limitations of standard interventions and exploit bodily information during cognitive rehabilitation. This systematic review aims to investigate whether 'multisensory technologies' compared to usual care treatment can be a viable alternative for cognitive rehabilitation. By applying PRISMA guidelines, we extracted data and assessed the bias of 10 studies that met the required criteria. We found that multisensory technologies were at least comparable to standard treatment but particularly effective for attention, spatial cognition, global cognition, and memory. Multisensory technologies consisted principally of virtual reality alone or combined with a motion tracking system. Multisensory technologies without motion tracking were more effective than standard procedures, whereas those with motion tracking showed balanced results for the two treatments. Limitations of the included studies regarded the population (e.g., no study on acute stroke), assessment (e.g., lack of multimodal/multisensory pre-post evaluation), and methodology (e.g., sample size, blinding bias). Recent advancements in technological development and metaverse open new opportunities to design embodied rehabilitative programs.

Piloting Virtual Reality Photo-Based Tours among Students of a Filipino Language Class: A Case of Emergency Remote Teaching in Japan

The State of Emergency declaration in Japan due to the COVID-19 pandemic affected many aspects of society in the country, much like the rest of the world. One sector that felt its disruptive impact was education. As educational institutions raced to implement emergency remote teaching (ERT) to continue providing the learning needs of students, some have opened to innovative interventions. This paper describes a case of ERT where Filipino vocabulary was taught to a class of Japanese students taking Philippine Studies in a Japanese university using a cognitive innovation based on virtual reality, an immersive technology often researched for immersion and presence. Students were divided into three groups to experience six lessons designed around virtual reality photo-based tours at different immersion levels. While the effect of immersion on satisfaction was not found to be statistically significant, presence and satisfaction were found to be correlated. Despite challenges that were encountered, benefits like enjoyment, increased engagement, and perceived learning were reported by the students. Our findings exemplify how emerging multisensory technologies can be used to enhance affective and cognitive dimensions of human experience while responding to gaps created by the spatial limitations of remote learning.

Digital Scent Technology: Toward the Internet of Senses and the Metaverse

Multisensory technologies have paved the way for the collective transfer and sharing of the senses on the Internet. The Human-Computer Interaction (HCI) community considers digital olfactory stimulation to be a helpful step in developing technology related to digital multisensory communication. This article provides a brief overview of digital scent technology through the recent electrical interfaces and past commercial efforts using chemical odors. We present the practical application of technology in the field of digital communication, marketing, and well being. We also discuss the commercial use of the Internet of Senses (IoS) and its cornerstone IMT-2030 (6G), contributing to the intimate nature of multisensory experiences from reality. Finally, we observe a symbiotic relationship between the future technologies of the IoS and the Metaverse, with 6G being the connecting link and the primary activator of the multisensory experiences.

Regenerative Virtual Therapy: The Use of Multisensory Technologies and Mindful Attention for Updating the Altered Representations of the Bodily Self.

The term “regenerative medicine” (RM) indicates an emerging trend in biomedical sciences that aims at replacing, engineering, or regenerating human cells, tissues, or organs to restore or establish normal function. So far, the focus of RM has been the physical body. Neuroscience, however, is now suggesting that mental disorders can be broadly characterized by a dysfunction in the way the brain computes and integrates the representations of the inner and outer body across time [bodily self-consciousness (BSC)]. In this perspective, we proposed a new kind of clinical intervention, i.e., “Regenerative Virtual Therapy” (RVT), which integrates knowledge from different disciplines, from neuroscience to computational psychiatry, to regenerate a distorted or faulty BSC. The main goal of RVT was to use technology-based somatic modification techniques to restructure the maladaptive bodily representations behind a pathological condition. Specifically, starting from a Bayesian model of our BSC (i.e., body matrix), we suggested the use of mindful attention, cognitive reappraisal, and brain stimulation techniques merged with high-rewarding and novel synthetic multisensory bodily experience (i.e., a virtual reality full-body illusion in sync with a low predictabIlity interoceptive modulation) to rewrite a faulty experience of the body and to regenerate the wellbeing of an individual. The use of RVT will also offer an unprecedented experimental overview of the dynamics of our bodily representations, allowing the reverse-engineering of their functioning for hacking them using advanced technologies.

Multisensory Integration as per Technological Advances: A Review.

Multisensory integration research has allowed us to better understand how humans integrate sensory information to produce a unitary experience of the external world. However, this field is often challenged by the limited ability to deliver and control sensory stimuli, especially when going beyond audio–visual events and outside laboratory settings. In this review, we examine the scope and challenges of new technology in the study of multisensory integration in a world that is increasingly characterized as a fusion of physical and digital/virtual events. We discuss multisensory integration research through the lens of novel multisensory technologies and, thus, bring research in human–computer interaction, experimental psychology, and neuroscience closer together. Today, for instance, displays have become volumetric so that visual content is no longer limited to 2D screens, new haptic devices enable tactile stimulation without physical contact, olfactory interfaces provide users with smells precisely synchronized with events in virtual environments, and novel gustatory interfaces enable taste perception through levitating stimuli. These technological advances offer new ways to control and deliver sensory stimulation for multisensory integration research beyond traditional laboratory settings and open up new experimentations in naturally occurring events in everyday life experiences. Our review then summarizes these multisensory technologies and discusses initial insights to introduce a bridge between the disciplines in order to advance the study of multisensory integration.

An Advanced Operation Mode with Product-Service System Using Lifecycle Big Data and Deep Learning

As a successful business strategy for enhancing environmental sustainability and decreasing the natural resource consumption of societies, the product-service system (PSS) has raised significant interests in the academic and industrial community. However, with the digitisation of the industry and the advancement of multisensory technologies, the PSS providers face many challenges. One major challenge is how the PSS providers can fully capture and efficiently analyse the operation and maintenance big data of different products and different customers in different conditions to obtain insights to improve their production processes, products and services. To address this challenge, a new operation mode and procedural approach are proposed for operation and maintenance of bigger cluster products, when these products are provided as a part of PSS and under exclusive control by the providers. The proposed mode and approach are driven by lifecycle big data of large cluster products and employs deep learning to train the neural networks to identify the fault features, thereby monitoring the products’ health status. This new mode is applied to a real case of a leading CNC machine provider to illustrate its feasibility. Higher accuracy and shortened time for fault prediction are realised, resulting in the provider’s saving of the maintenance and operation cost.

ColorWatch: Color Perceptual Spatial Tactile Interface for People with Visual Impairments

Tactile perception enables people with visual impairments (PVI) to engage with artworks and real-life objects at a deeper abstraction level. The development of tactile and multi-sensory assistive technologies has expanded their opportunities to appreciate visual arts. We have developed a tactile interface based on the proposed concept design under considerations of PVI tactile actuation, color perception, and learnability. The proposed interface automatically translates reference colors into spatial tactile patterns. A range of achromatic colors and six prominent basic colors with three levels of chroma and values are considered for the cross-modular association. In addition, an analog tactile color watch design has been proposed. This scheme enables PVI to explore artwork or real-life object color by identifying the reference colors through a color sensor and translating them to the tactile interface. The color identification tests using this scheme on the developed prototype exhibit good recognition accuracy. The workload assessment and usability evaluation for PVI demonstrate promising results. This suggest that the proposed scheme is appropriate for tactile color exploration.

Analysis directions for the development of future telecommunication technologies

The sixth generation (6G) of mobile networks is expected to be implemented in the next decade. The development of forecasts, applications, technologies and 6G standards has already become a popular research topic. The article analyzes new services and applications, such as holographic media programs, multisensory technologies. Basic technological solutions that solve the problems of existing fifth generation (5G) mobile networks are analyzed. These limitations include data rates, latency, reliability, availability, processing, connection density, and global coverage. Highlighted social and technological trends that initiate the movement to 6G. The most modern researches, technical problems which promote development of telecommunication technologies are discussed. The requirements and key technologies required for the implementation of 6G applications are presented in detail.

Actitudes del profesorado sobre la innovación con herramientas TIC multisensoriales en entornos inclusivos

There is evidence that the use of digital games based on multisensory tools provides benefits in different dimensions of learning for students with special educational needs. However, the use of these tools in early childhood and primary education classrooms to promote the inclusion of such students has not been treated in depth. The aim of this paper is to analyze how the participation of a group of teachers in a professional development program focused on the integration in the classroom of games based on multisensory technologies contributed to the evolution of their attitudes about the adoption of these technologies in school contexts from an inclusive perspective. In this research, a mixed method with a sequential explanatory strategy has been used. The techniques and instruments used, interviews and questionnaires, have allowed us to collect data on the evolution of the concerns that arose after the training received in the framework of the European project INTELed. The results show that the teachers highly valued the collaborations that emerged during the project, and that they experienced an evolution towards higher levels of concern regarding sustainability and transfer in this type of innovations in the classroom.

From “touch” to a “multisensory” experience: The impact of technology interface and product type on consumer responses

AbstractOnline retailers are increasingly using augmented reality (AR) and virtual reality (VR) technologies to solve mental and physical intangibility issues in a product evaluation. Moreover, the technologies are easily available and accessible to consumers via their smartphones. The authors conducted three experiments to examine consumer responses to technology interfaces (AR/VR and mobile apps) for hedonic and utilitarian products. The results show that AR is easier to use (vs. app), and users find AR more responsive when buying a hedonic (vs. utilitarian) product. Touch interface users are likely to have a more satisfying experience and greater recommendation intentions, as compared with AR, for buying utilitarian products. In contrast, a multisensory environment (AR) results in a better user experience for purchasing a hedonic product. Moreover, multisensory technologies lead to higher visual appeal, emotional appeal, and purchase intentions. The research contributes to the literature on computer‐mediated interactions in a multisensory environment and proposes actionable recommendations to online marketers.

Multisensory Interactive Technologies for Primary Education: From Science to Technology.

While technology is increasingly used in the classroom, we observe at the same time that making teachers and students accept it is more difficult than expected. In this work, we focus on multisensory technologies and we argue that the intersection between current challenges in pedagogical practices and recent scientific evidence opens novel opportunities for these technologies to bring a significant benefit to the learning process. In our view, multisensory technologies are ideal for effectively supporting an embodied and enactive pedagogical approach exploiting the best-suited sensory modality to teach a concept at school. This represents a great opportunity for designing technologies, which are both grounded on robust scientific evidence and tailored to the actual needs of teachers and students. Based on our experience in technology-enhanced learning projects, we propose six golden rules we deem important for catching this opportunity and fully exploiting it.

Zelige Door on Golborne Road

In this paper I discuss the multisensory digital interface and art installation Zelige Door on Golborne Road as part of the wider research project ‘Mapping Memory Routes: Eliciting Culturally Diverse Collective Memories for Digital Archives’. The interface is conceived as a tool for capturing and displaying the living heritage of members of Moroccan migrant communities, shared through an artwork composed of a digital interactive sensorial map of Golborne Road (also known as Little Morocco), which includes physical objects related to various aspects of Moroccan culture, each requiring a different sense to be experienced (smell, taste, sight, hearing, touch). Augmented Reality (AR) and olfactory technologies have been used in the interface to superimpose pre-recorded video material and smells to the objects. As a result, the neighbourhood is represented as a living museum of cultural memories expressed in the form of artefacts, sensory stimulation and narratives of citizens living, working or visiting the area. Based on a model I developed for the multisensory installation ‘Streets of...7 cities in 7 minutes’, the interface was designed with Dr Mariza Dima (HCI designer), and Prof. Monica Bordegoni and Dr Marina Carulli (olfactory technology engineers/designers) to explore new methods able to elicit cultural Collective Memories through the use of multi-sensory technologies. The tool is also aimed at stimulating collective curatorial practices and democratise decision-making processes in urban planning and cultural heritage.

Multisensory Technology for Flavor Augmentation: A Mini Review.

There is growing interest in the development of new technologies that capitalize on our emerging understanding of the multisensory influences on flavor perception in order to enhance human–food interaction design. This review focuses on the role of (extrinsic) visual, auditory, and haptic/tactile elements in modulating flavor perception and more generally, our food and drink experiences. We review some of the most exciting examples of recent multisensory technologies for augmenting such experiences. Here, we discuss applications for these technologies, for example, in the field of food experience design, in the support of healthy eating, and in the rapidly growing world of sensory marketing. However, as the review makes clear, while there are many opportunities for novel human–food interaction design, there are also a number of challenges that will need to be tackled before new technologies can be meaningfully integrated into our everyday food and drink experiences.