Multisensory Devices Research Articles

Fabrication of carbon nanotube neuromorphic thin film transistor arrays and their applications for flexible olfactory-visual multisensory synergy recognition

Abstract Artificial multisensory devices play a key role for the human-computer interaction in the field of artificial intelligence (AI). In this work, we have designed and constructed a novel olfactory-visual bimodal neuromorphic carbon nanotube thin film transistor (TFT) arrays for artificial olfactory-visual multisensory synergy recognition with the ultralow single-pulse power consumption of 25 aJ, employing semiconducting single-walled carbon nanotubes (sc-SWCNTs) as channel materials and gas sensitive materials, and poly[[4,8-bis[5-(2-ethylhexyl)-2-thienyl]benzo[1,2-b:4,5-b0]dithiophene-2,6-diyl]-2,5-thiophenediyl-[5,7-bis(2-ethylhexyl)-4,8-dioxo-4H,8H-benzo[1,2-c:4,5-c0]dithio-phene-1,3-diyl]] (PBDB-T) as the photosensitive material. It is noted that it is the first time to realize the simulation of olfactory and visual senses (from 280 nm to 650 nm) with the wide operating temperature range (0-150 °C) in a single SWCNT TFT device and successfully simulate the recovery of olfactory senses after COVID-19 by olfactory-visual synergy. Furthermore, our SWCNT neuromorphic TFT devices with high Ion/Ioff ratio (up to 106) at a low operating voltage (-2-0.5 V) can mimic not only the basic biological synaptic functions of olfaction and vision (such as paired-pulse facilitation, short-term plasticity, and long-term plasticity), but also optical wireless communication by Morse code. The proposed multisensory, broadband light-responsive, low-power synaptic devices provide great potential for developing AI robots to face complex external environments.

A Deep Learning-Based Platform for Workers' Stress Detection Using Minimally Intrusive Multisensory Devices.

The advent of Industry 4.0 necessitates substantial interaction between humans and machines, presenting new challenges when it comes to evaluating the stress levels of workers who operate in increasingly intricate work environments. Undoubtedly, work-related stress exerts a significant influence on individuals' overall stress levels, leading to enduring health issues and adverse impacts on their quality of life. Although psychological questionnaires have traditionally been employed to assess stress, they lack the capability to monitor stress levels in real-time or on an ongoing basis, thus making it arduous to identify the causes and demanding aspects of work. To surmount this limitation, an effective solution lies in the analysis of physiological signals that can be continuously measured through wearable or ambient sensors. Previous studies in this field have mainly focused on stress assessment through intrusive wearable systems susceptible to noise and artifacts that degrade performance. One of our recently published papers presented a wearable and ambient hardware-software platform that is minimally intrusive, able to detect human stress without hindering normal work activities, and slightly susceptible to artifacts due to movements. A limitation of this system is its not very high performance in terms of the accuracy of detecting multiple stress levels; therefore, in this work, the focus was on improving the software performance of the platform, using a deep learning approach. To this purpose, three neural networks were implemented, and the best performance was achieved by the 1D-convolutional neural network with an accuracy of 95.38% for the identification of two levels of stress, which is a significant improvement over those obtained previously.

An evaluation of recent advancements in biological sensory organ-inspired neuromorphically tuned biomimetic devices.

In the field of neuroscience, significant progress has been made regarding how the brain processes information. Unlike computer processors, the brain comprises neurons and synapses instead of memory blocks and transistors. Despite advancements in artificial neural networks, a complete understanding concerning brain functions remains elusive. For example, to achieve more accurate neuron replication, we must better understand signal transmission during synaptic processes, neural network tunability, and the creation of nanodevices featuring neurons and synapses. This study discusses the latest algorithms utilized in neuromorphic systems, the production of synaptic devices, differences between single and multisensory gadgets, recent advances in multisensory devices, and the promising research opportunities available in this field. We also explored the ability of an artificial synaptic device to mimic biological neural systems across diverse applications. Despite existing challenges, neuroscience-based computing technology holds promise for attracting scientists seeking to enhance solutions and augment the capabilities of neuromorphic devices, thereby fostering future breakthroughs in algorithms and the widespread application of cutting-edge technologies.

Intelligent ADL Recognition via IoT-Based Multimodal Deep Learning Framework.

Smart home monitoring systems via internet of things (IoT) are required for taking care of elders at home. They provide the flexibility of monitoring elders remotely for their families and caregivers. Activities of daily living are an efficient way to effectively monitor elderly people at home and patients at caregiving facilities. The monitoring of such actions depends largely on IoT-based devices, either wireless or installed at different places. This paper proposes an effective and robust layered architecture using multisensory devices to recognize the activities of daily living from anywhere. Multimodality refers to the sensory devices of multiple types working together to achieve the objective of remote monitoring. Therefore, the proposed multimodal-based approach includes IoT devices, such as wearable inertial sensors and videos recorded during daily routines, fused together. The data from these multi-sensors have to be processed through a pre-processing layer through different stages, such as data filtration, segmentation, landmark detection, and 2D stick model. In next layer called the features processing, we have extracted, fused, and optimized different features from multimodal sensors. The final layer, called classification, has been utilized to recognize the activities of daily living via a deep learning technique known as convolutional neural network. It is observed from the proposed IoT-based multimodal layered system's results that an acceptable mean accuracy rate of 84.14% has been achieved.

A Single Electronic Tattoo for Multisensory Integration.

Wearable electronics are garnering growing interest in various emerging fields including intelligent sensors, artificial limbs, and human-machine interfaces. A remaining challenge is to develop multisensory devices that can conformally adhere to the skin even during dynamic-moving environments. Here, a single electronic tattoo (E-tattoo) based on a mixed-dimensional matrix network, which integrates two-dimensional MXene nanosheets and one-dimensional cellulose nanofibers/Ag nanowires, is presented for multisensory integration. The multidimensional configurations endow the E-tattoo with excellent multifunctional sensing capabilities including temperature, humidity, in-plane strain, proximity, and material identification. In addition, benefiting from the satisfactory rheology of hybrid inks, the E-tattoos are able to be fabricated through multiple facile strategies including direct writing, stamping, screen printing, and three-dimensional printing on various hard/soft substrates. Especially, the E-tattoo with excellent triboelectric properties also can serve as a power source for activating small electronic devices. It is believed that these skin-conformal E-tattoo systems can provide a promising platform for next-generation wearable and epidermal electronics.

Deep Ontology-Based Human Locomotor Activity Recognition System via Multisensory Devices

Deep Ontology-Based Human Locomotor Activity Recognition System via Multisensory Devices

Multi-sensing properties of hybrid filled natural rubber nanocomposites using impedance spectroscopy

In this work, natural rubber nanocomposites filled with carbon black and carbon nanotubes were prepared via latex mixing, and their applicability as multisensory devices were analyzed through Electrochemical Impedance Spectroscopy (EIS). Static and cyclic strain sensing was performed until 300% strain and the piezoresistivity, piezo-impedance, and piezo-capacitance were evaluated. The results indicated that impedance magnitude could be used instead of calculated resistivity, especially in lower and medium strain ranges. In higher strains, the capacitive behavior turns predominant, and the piezo-capacitance becomes the ideal parameter to be measured providing higher sensitivities as the gage factor (GF) reaches ∼41.82 in 300% strain. The solvent sensing also showed that piezo-capacitance is a better measure than piezo-impedance, with a quicker response time and greater sensibility. This study demonstrated that using EIS in strain and solvent sensing of rubber nanocomposites can offer more detailed information about complex transport mechanisms and improve the sensitivity, linearity, and response time of elastomeric sensors.

Immersive Learning Research from SVR Publications: A Re-conduction of the Systematic Mapping Study

Immersive Learning (iL) is known as a recent area of research that uses three-dimensional virtual environments and multi-sensory devices, also known as immersive technologies, to support the improvement of learning outcomes. This work aims to obtain evidence of theoretical and technological aspects of iL from the Symposium on Virtual and Augmented Reality (SVR) publications. A Systematic Literature Mapping protocol was developed and executed in order to select the primary studies to perform the analysis and data extraction. 76 primary studies helped to answer the research questions. A large part of the contributions by the SVR community are virtual environments that support education in the health area. In addition, some gaps and research opportunities were identified: virtual environments that serve audiences with special needs; development frameworks that consider pedagogical aspects; the use of biometric measures to support the validation of improved learning outcomes and more flexible tools that support educators in the development of immersive virtual environments that do not require specific knowledge in coding or 3D modeling.

A Multi-Sensor Data-Fusion Method Based on Cloud Model and Improved Evidence Theory.

The essential factors of information-aware systems are heterogeneous multi-sensory devices. Because of the ambiguity and contradicting nature of multi-sensor data, a data-fusion method based on the cloud model and improved evidence theory is proposed. To complete the conversion from quantitative to qualitative data, the cloud model is employed to construct the basic probability assignment (BPA) function of the evidence corresponding to each data source. To address the issue that traditional evidence theory produces results that do not correspond to the facts when fusing conflicting evidence, the three measures of the Jousselme distance, cosine similarity, and the Jaccard coefficient are combined to measure the similarity of the evidence. The Hellinger distance of the interval is used to calculate the credibility of the evidence. The similarity and credibility are combined to improve the evidence, and the fusion is performed according to Dempster’s rule to finally obtain the results. The numerical example results show that the proposed improved evidence theory method has better convergence and focus, and the confidence in the correct proposition is up to 100%. Applying the proposed multi-sensor data-fusion method to early indoor fire detection, the method improves the accuracy by 0.9–6.4% and reduces the false alarm rate by 0.7–10.2% compared with traditional and other improved evidence theories, proving its validity and feasibility, which provides a certain reference value for multi-sensor information fusion.

Combining cardiac monitoring with actigraphy aids nocturnal arousal detection during ambulatory sleep assessment in insomnia.

Study ObjectivesThe objective assessment of insomnia has remained difficult. Multisensory devices collecting heart rate (HR) and motion are regarded as the future of ambulatory sleep monitoring. Unfortunately, reports on altered average HR or heart rate variability (HRV) during sleep in insomnia are equivocal. Here, we evaluated whether the objective quantification of insomnia improves by assessing state-related changes in cardiac measures.MethodsWe recorded electrocardiography, posture, and actigraphy in 33 people without sleep complaints and 158 patients with mild to severe insomnia over 4 d in their home environment. At the microscale, we investigated whether HR changed with proximity to gross (body) and small (wrist) movements at nighttime. At the macroscale, we calculated day-night differences in HR and HRV measures. For both timescales, we tested whether outcome measures were related to insomnia diagnosis and severity.ResultsAt the microscale, an increase in HR was often detectable already 60 s prior to as well as following a nocturnal chest, but not wrist, movement. This increase was slightly steeper in insomnia and was associated with insomnia severity, but future EEG recordings are necessary to elucidate whether these changes occur prior to or simultaneously with PSG-indicators of wakefulness. At the macroscale, we found an attenuated cardiac response to sleep in insomnia: patients consistently showed smaller day-night differences in HR and HRV.ConclusionsIncorporating state-related changes in cardiac features in the ambulatory monitoring of sleep might provide a more sensitive biomarker of insomnia than the use of cardiac activity averages or actigraphy alone.

Effect of Tactile Masking on Multi-Sensory Haptic Perception.

Multi-sensory wearable haptic devices are able to encode a variety of information using multiple haptic cues. However, simultaneous cues can be misperceived due to tactile masking effects. In this paper, we investigate the effect of masking on the perception of skin stretch and squeeze. We performed three experiments measuring the just-noticeable difference (JND) and the absolute threshold of skin stretch and squeeze alone and in the presence of simultaneous haptic cues. Additionally, we investigate the relative perceptual amplitudes of these haptic cues. Results indicate that the JND for a skin stretch cue increases with a masking squeeze cue, while the JND for a squeeze cue does not change with a masking stretch cue. Also, masking has a significant effect on the absolute threshold of both skin stretch and squeeze. These results suggest that the effect of masking diminishes as haptic cues become larger in amplitude. The results from the subjective equality experiment suggest a potential nonlinear relationship between perceptual magnitudes. Further testing should be carried out to investigate this relationship. Future multi-sensory devices can use these perceptual experiment findings to ensure the delivery of salient cues to users.

Investigating how we read translations

AbstractSince its inception, Translation Studies has hinged on theoretical concepts of effects and reception, with various reader-oriented notions such as equivalent effect,skopos, acceptability and adequacy, and user-centredness, to name but a few, having pervaded the discipline for decades. Despite this preoccupation with the phenomenology of translations, we still know very little about how translations are actually experienced –writtentranslations especially. This article calls for an expansion of research into the reception and experience of source texts and their translations, reviewing the opportunities afforded by recent technological developments in eye-tracking, galvanic skin response sensors, echocardiogram monitors, and other multi-sensory devices. Using a short case study, a number of research questions and an outline of an experimental method are proposed to contrast the reading experience of two translations of the same source text, serving as a prompt for future research of this kind. By drawing inspiration from the few existing examples of research in this incipient paradigm and the considerations offered in the example, this article aims to stimulate future research to explore the vast untapped potential in this area and to arrive at a better understanding of the effects that different translation approaches yield and the potential variation in effects between source and target text.

Past, Present, and Future of Multisensory Wearable Technology to Monitor Sleep and Circadian Rhythms.

Movement-based sleep-wake detection devices (i.e., actigraphy devices) were first developed in the early 1970s and have repeatedly been validated against polysomnography, which is considered the “gold-standard” of sleep measurement. Indeed, they have become important tools for objectively inferring sleep in free-living conditions. Standard actigraphy devices are rooted in accelerometry to measure movement and make predictions, via scoring algorithms, as to whether the wearer is in a state of wakefulness or sleep. Two important developments have become incorporated in newer devices. First, additional sensors, including measures of heart rate and heart rate variability and higher resolution movement sensing through triaxial accelerometers, have been introduced to improve upon traditional, movement-based scoring algorithms. Second, these devices have transcended scientific utility and are now being manufactured and distributed to the general public. This review will provide an overview of: (1) the history of actigraphic sleep measurement, (2) the physiological underpinnings of heart rate and heart rate variability measurement in wearables, (3) the refinement and validation of both standard actigraphy and newer, multisensory devices for real-world sleep-wake detection, (4) the practical applications of actigraphy, (5) important limitations of actigraphic measurement, and lastly (6) future directions within the field.

Le maître dans l’œil du disciple. À propos de l’apprentissage ou de l’initiation écologique dans Les Neuf consciences du Malfini de Patrick Chamoiseau

Les Neuf consciences du Malfini de Patrick Chamoiseau est un roman initiatique articulé autour de l’expérience pédagogique d’un disciple et des prouesses didactico-pédagogiques d’un maître. Fondée sur l’observation minutieuse et l’imitation formatrice du maître, cet apprentissage permet à l’initié de transcender ses élans prédateurs pour accéder au stade de la responsabilité écologique. Contrairement au roman de formation traditionnel dans lequel le maître mobilise le dispositif multisensoriel et multimodal pour relayer le savoir ou la vérité, celui-ci met le lecteur devant une situation « pédagogique » archétypale où la relation maître-disciple n’est pas interpersonnelle, interactive, volontaire ou ritualisée. En outre, le lien pédagogique ne s’y déploie pas seulement entre les protagonistes mis en texte, mais aussi entre le roman et le lecteur qui en est la cible privilégiée. À l’aune de l’écocritique et de l’approche sémiologique du personnage, cette réflexion se propose d’étudier le processus d’initiation et la dynamique des relations maître-disciple pour montrer que, par ce récit allégorique, Chamoiseau fait le procès de l’anthropocentrisme comme cause principale du déclin écologique et préconise l’horizontalité dans les relations Homme/Nature, gage de l’équilibre écologique.

Management of daily physical activity and diabetic foot prevention

Even if physical activity plays a key role within diabetic foot treatment its use and the results obtained from this treatment seem to be still limited. Nowadays, new and even more advanced technologies for the long term daily physical activity monitoring are available and they are radically changing some aspects of physical activity such as its amount, features and monitoring. In spite of the past, the several electronic devices that are currently available can be integrated into routine care and provide essential information for management to both the healthcare providers and patients. In particular, since the end of the last century, an increasing number of studies have applied the movement monitoring to patients at risk or with history of ulceration. The questionnaires have been progressively replaced with modern technologies such as accelerometers or complex multisensory devices able to objectively measure the physical activity performed. The data collected through the use of such devices can allow a better assessment of patient’s condition and provide useful information for the definition of a more complete treatment protocol. Daily physical activity monitoring devices provide to the Diabetes Units information on the typology, quantity, distribution and intensity of the daily physical activity performed by each patient concurring to the prevention of foot ulcers that represent the most dreadful diabetes complications. The different functions and modes of operation of monitoring devices can be integrated to provide a more comprehensive and intelligent monitoring system that provide valuable information on patients’ ongoing health status and the physical activity performed during daily life. These devices can manage in real time or even in remote the physical activity performed in addition to calculate that to be performed in the following hours. As a result, they contribute to improve patients’ lifestyle and reduce the costs for the treatment of such complications. The aim of this review is to define and emphasize the role of a long term daily physical activity monitoring in the prevention of diabetic foot ulcers.

Cyber-Physical System Framework for Measurement and Analysis of Physical Activities

Several recent studies in Cyber-Physical Systems (CPS) focus on monitoring human movement and capturing data for further processing and analysis. However, there is a lack of studies that address the configurability and modularity of these systems, which is important for designing customized systems with customized devices. We propose a solution to solve this through a modular framework that automatically recognizes and configures new devices and provides real-time data wirelessly. The proposed framework creates a Digital Twin of the physical device and mirrors its attributes and sensory information into the cyber world so they can be used in real-time and post-routine analysis. As a proof of concept, a configurable CPS model for physical activities monitoring is designed and implemented. The designed gait monitoring and analysis system delivers spatiotemporal data from multiple multi-sensory devices to a central data handling and backup cloud server over conventional IEEE802.11 Wi-Fi. An experiment involving a young athlete examined whether or not the CPS components would recognize each other over foreign networks and communicate accurate information.

Arranging sensations: smell and taste in augmented and virtual reality

ABSTRACTThe development of digital taste and smell underscores the importance of cultural dimensions of bodily perception in augmented reality (AR) and virtual reality (VR) devices. This can be seen in Vocktail and Season Traveller, two digital devices incorporating taste and smell. Vocktail is an AR technology that augments the experience of drinking water, or even air, through the electrical stimulation of tastebuds and the manipulation of color and smell. Season Traveller is a VR game in which the user moves through four seasonal landscapes. It uses wind, odor, and temperature in addition to the more standard audio-visual displays. The cultural dimensions of these devices can be examined using phenomenological terms. They instigate perceptual circuits, and call on and create sedimented habits. Although VR and AR are often thought of in terms of their similitude to reality, understanding Vocktail and Season Traveller this way illustrates the world-creating dimension of multisensory devices. These technologies structure and shift thresholds of taste and smell, reworking past perceptual styles and habits to develop new perceptual experiences. In so doing, Season Traveller and Vocktail throw to the fore questions about the conditions according to which people exercise their senses in digitally dominated environments.

A cloud-based near real-time slope movement monitoring system

A cloud-based near real-time data management system for geotechnical monitoring that can efficiently record, transfer, store, manage, process, analyse and evaluate monitoring data and generate easy to view and use information is presented. The database structure and manual of the system functionalities for addressing issues during data upload and processing, record handling for multisensory devices, data visualisation using common reference dates, data error correction and parametric handling of import files are presented in detail. New features that are in development include a more robust warning system and a journaling system for visual observations. The system can reliably and quickly generate alarms and information about excessive movements with respect to mining and other operations that require monitoring.

Culture in multisensory computing: dividing the body for a digital future

ABSTRACTThis paper examines the ways in which computer science and a selection of multisensory digital devices modulate the term culture. Three self-identified, ‘cultural computing’ devices are examined: ZENetic, Alice’s Adventures and GRIOT’S Japanese Renku. The devices variously configure relationships between bodies and culture so that the body is thought to provide a window into particular cultures, as well as a universal tool of their transmission. These delineations of the term culture occur in continuity with surrounding political histories and projects, including racial and ethnic ones. Dividing the body between the extra cultural and the culturally specific works to secure the communicability as well as the exclusivity of cultural practices. It is an instance of what Merleau-Ponty terms a divergence, a gap within the body between two imbricated parts. In this way, these devices present culture as a source of innovation that combats global and abstracted computing practices.

Calibrating chemical multisensory devices for real world applications: An in-depth comparison of quantitative machine learning approaches

Chemical multisensor devices need calibration algorithms to estimate gas concentrations. Their possible adoption as indicative air quality measurements devices poses new challenges due to the need to operate in continuous monitoring modes in uncontrolled environments. Several issues, including slow dynamics, continue to affect their real world performances. At the same time, the need for estimating pollutant concentrations on board the devices, especially for wearables and IoT deployments, is becoming highly desirable. In this framework, several calibration approaches have been proposed and tested on a variety of proprietary devices and datasets; still, no thorough comparison is available to researchers. This work attempts a benchmarking of the most promising calibration algorithms according to recent literature with a focus on machine learning approaches. We test the techniques against absolute and dynamic performances, generalization capabilities and computational/storage needs using three different datasets sharing continuous monitoring operation methodology. Our results can guide researchers and engineers in the choice of optimal strategy. They show that non-linear multivariate techniques yield reproducible results, outperforming linear approaches. Specifically, the Support Vector Regression method consistently shows good performances in all the considered scenarios. We highlight the enhanced suitability of shallow neural networks in a trade-off between performance and computational/storage needs. We confirm, on a much wider basis, the advantages of dynamic approaches with respect to static ones that only rely on instantaneous sensor array response. The latter have been shown to be best choice whenever prompt and precise response is needed.