Haptic Data Research Articles

AI solutions for overcoming delays in telesurgery and telementoring to enhance surgical practice and education

AbstractArtificial intelligence (AI) has emerged as a transformative tool in surgery, particularly in telesurgery and telementoring. However, its potential to enhance data transmission efficiency and reliability in these fields remains unclear. While previous reviews have explored the general applications of telesurgery and telementoring in specific surgical contexts, this review uniquely focuses on AI models designed to optimise data transmission and mitigate delays. We conducted a comprehensive literature search on PubMed and IEEE Xplore for studies published in English between 2010 and 2023, focusing on AI-driven, surgery-related, telemedicine, and delay-related research. This review includes methodologies from journals, conferences, and symposiums. Our analysis identified a total of twelve AI studies that focus on optimising network resources, enhancing edge computing, and developing delay-robust predictive applications. Specifically, three studies addressed wireless network resource optimisation, two proposed low-latency control and transfer learning algorithms for edge computing, and seven developed delay-robust applications, five of which focused on motion data, with the remaining two addressing visual and haptic data. These advancements lay the foundation for a truly holistic and context-aware telesurgical experience, significantly transforming remote surgical practice and education. By mapping the current role of AI in addressing delay-related challenges, this review highlights the pressing need for collaborative research to drive the evolution of telesurgery and telementoring in modern robotic surgery.

A provably secure and practical end-to-end authentication scheme for tactile Industrial Internet of Things

In the Industrial Internet of Things (IIoT), haptic control of machines or robots can be managed remotely. However, with the emergence of Tactile Industrial Internet of Things (TIIoT), the transmission of haptic data over public channels has raised security and privacy concerns. In such an environment, mutual authentication between haptic users and remotely controlled entities is crucial to prevent illegal control by adversaries. Therefore, we propose an end-to-end authentication scheme, SecTIIoT, to establish secure communication between haptic users and remote IoT devices. The scheme addresses security issues by using lightweight hash cryptographic primitives and employs a useful piggyback strategy to improve authentication efficiency. We demonstrate that SecTIIoT is resilient to various known attacks with formal security proofs and informal security analysis. Furthermore, our detailed performance analysis shows that SecTIIoT outperforms existing lightweight authentication schemes as it provides more security features while reducing computation and communication costs.

SURG-01. DIFFERENTIATION OF GLIOBLASTOMA AND MALIGNANT MENINGIOMA FROM NORMAL BRAIN BY TISSUE STIFFNESS UTILIZING MICROSURGICAL FORCEPS EQUIPPED WITH HAPTICS TECHNOLOGY

Abstract Stiffness of human cancers strongly correlate with their pathology, and it is reported that stiffness can be used as a biomarker. Furthermore, comprehensive analysis has suggested the usefulness of cancer stiffness in diagnosis and malignancy prediction and its association with molecular expression, as well as its potential for predicting and diagnosing postoperative complications. Neurosurgeons perform tumor resection relying on the sensations that they feel with their own fingertips (tactile sensation), but it takes years of surgical experience to accurately distinguish brain tumor from the surrounding normal brain tissue. Haptics is a new technology regarding touching sensation. Especially, our technology, real haptics, can communicate, record, and reproduce the real sensation. It is also possible to measure and analyze the physical property such as stiffness of the touching object from the obtained haptic data. We developed a surgical forceps equipped with haptics technology to distinguish brain tumor tissue from surrounding normal brain tissue. three glioblastoma cell lines (SF126, U87, U251) and a malignant meningioma cell line (IOMM-Lee) were each transplanted intracranially into 5 nude mice. Tumor size was periodically monitored and mice were euthanized at appropriate time points. The stiffness of the tumor and normal brain tissue was measured using a master-slave integrated surgical forceps prototype. The results statistically showed that all four types of brain tumor tissues were stiffer than normal brain tissues. (p = 0.001) In addition, malignant meningioma was statistically stiffer than glioblastomas. (p = 0.025) This result is consistent with actual surgical sensations. We have shown that our surgical forceps has the ability to distinguish pathology of brain tumors as well as to distinguish brain tumor from normal brain tissue. The forceps is also helpful for the neurosurgeons to sense minute change in tissue stiffness.

A Long Short-Term Memory-Based Model for Kinesthetic Data Reduction

This paper proposes a novel mathematical model for teleoperation over communication networks. For teleoperation over a communication network, a high packet rate can result in inefficient data transmission and cross-traffic problems, leading to extra delay and jitter. This paper proposes an LSTM-based mathematical model which focuses on kinesthetic data reduction without loss of transparency during the transmission process through joint training combined with haptic data and perceptual deadband. Since the LSTM network can deal with a time series of haptic data, we further test the system performance through practically collected data. We investigate the packet rate and perceptual transparency of the proposed mathematical model by comparing with the conventional deadband. Additionally, we compare the proposed mathematical model with the perceptual deadband-based codecs. Simulation results show that the proposed solution further reduces the packet rate when dealing with haptic data without noticeable distortion. Also, comparing with the current JND perceptual threshold, the proposed mathematical model helps improve the practicality of the bilateral teleoperation system without losing transparency.

Conception And Implementation of an virtual Reality application for the evaluation of different types of commercially available haptic gloves

Abstract Immersion and presence are important aspects of virtual reality (VR). Efforts are therefore being made to enhance these effects in order to increase the impact of VR and its transferability to reality. This is particularly important in research areas such as medical education and psychology. The development of haptic gloves has therefore increased significantly in recent years. These are immersive input devices designed to make virtual objects tangible. However, the research field is still very new and there is insufficient research on their handling and effect. For this reason, an application was developed to compare different haptic data gloves with different haptic technologies. A glove with vibrotactile feedback (Manus Prime X Haptic VR) and a glove with additional force feedback (SenseGlove) were used and compared in multiple tasks. These include finger tracking and gesture recognition, as well as hand tracking and the behaviour of gloves when grasping objects with and without haptics. The aim is to determine how well the haptics could be integrated and which is more accepted by the test subjects. For this purpose, a questionnaire has been designed to evaluate the effect of the haptic gloves on users. It aims to get a first impression of the tactile gloves available on the market and to advance research in the area.

Emulation Framework for Haptic Data Transmission Using Real-Time Transport Protocol

The Tactile Internet (TI) can be regarded as the next evolution in the world of communication. With its envisioned purpose and potential in shaping up the economy, industry and society, this paradigm aims to bring a new dimension to life by enabling humans to interact with machines remotely and in real-time with haptic and kinesthetic feedback. However, to translate this into reality, Tactile Internet will need to meet the stringent requirements of extremely low latency in conjunction with ultra-high reliability, availability, and security. This poses a challenge on the available communication systems to achieve a round-trip delay within 1 to 10 milliseconds time bound that enables the timely delivery of critical tactile and haptic sensations.
 This paper aims to evaluate the Real-Time Transport Protocol (RTP) through an emulation framework. It integrates containerization using Linux-based Docker Containers with NS-3 Network Simulator to conceptualize a haptic teleoperation system. The framework is then used to test the protocol’s feasibility for delivering texture haptic data between master and slave domains in accordance with the end-to-end delay requirements specified by IEEE 1918.1 standards. The results have shown that the timely provision of haptic data is achievable by obtaining an average round-trip delay of 17.8493 ms from the emulation experiment. As such, the results satisfy the expected IEEE 1918.1 standards constraints for medium-dynamic environment use cases.

Surface Material Classification Based on Unbalanced Visual and Haptic Data: A Double-DQN Method

The paper focuses on surface material classification with unbalanced visual and haptic data, which is important in teleoperation and robotic recognition. For this problem, existent classification methods inevitably suffer from performance degradation, as they tend to emphasize the major classes and ignore the minor ones. To overcome such an issue, we address this classification problem by the double deep Q-learning network (DDQN) method which not only offers strong representation ability, but also avoids over-estimation. Specifically, we first transform haptic accelerations to their spectrograms by Short-Time Fourier Transform (STFT), and respectively feed visual images and haptic spectrograms to pre-trained ResNet50 for extracting low-dimensional feature vectors. Then, the hybrid visual-haptic feature vectors are fed into DDQN as a sequence of states. With respect to each state, DDQN assigns it with an estimated class label. By comparing with the true label, DDQN earns a reward that is dependent on the imbalance ratio of the class. Through maximization of the cumulative rewards, DDQN enhances its classification performance. For the purposes of validation, numerical evaluations are carried out on TUM69, LMT108 and HaTT datasets. The results show that DDQN outperforms existent methods in both classification performance and computational complexity.

Assessment of EEG-based functional connectivity in response to haptic delay.

Haptic technologies enable users to physically interact with remote or virtual environments by applying force, vibration, or motion via haptic interfaces. However, the delivery of timely haptic feedback remains a challenge due to the stringent computation and communication requirements associated with haptic data transfer. Haptic delay disrupts the realism of the user experience and interferes with the quality of interaction. Research efforts have been devoted to studying the neural correlates of delayed sensory stimulation to better understand and thus mitigate the impact of delay. However, little is known about the functional neural networks that process haptic delay. This paper investigates the underlying neural networks associated with processing haptic delay in passive and active haptic interactions. Nineteen participants completed a visuo-haptic task using a computer screen and a haptic device while electroencephalography (EEG) data were being recorded. A combined approach based on phase locking value (PLV) functional connectivity and graph theory was used. To assay the effects of haptic delay on functional connectivity, we evaluate a global connectivity property through the small-worldness index and a local connectivity property through the nodal strength index. Results suggest that the brain exhibits significantly different network characteristics when a haptic delay is introduced. Haptic delay caused an increased manifestation of the small-worldness index in the delta and theta bands as well as an increased nodal strength index in the middle central region. Inter-regional connectivity analysis showed that the middle central region was significantly connected to the parietal and occipital regions as a result of haptic delay. These results are expected to indicate the detection of conflicting visuo-haptic information at the middle central region and their respective resolution and integration at the parietal and occipital regions.

A Timestamp-Independent Haptic–Visual Synchronization Method for Haptic-Based Interaction System

The booming haptic data significantly improve the users’ immersion during multimedia interaction. As a result, the study of a Haptic-based Interaction System has attracted the attention of the multimedia community. To construct such a system, a challenging task is the synchronization of multiple sensorial signals that is critical to the user experience. Despite audio-visual synchronization efforts, there is still a lack of a haptic-aware multimedia synchronization model. In this work, we propose a timestamp-independent synchronization for haptic–visual signal transmission. First, we exploit the sequential correlations during delivery and playback of a haptic–visual communication system. Second, we develop a key sample extraction of haptic signals based on the force feedback characteristics and a key frame extraction of visual signals based on deep-object detection. Third, we combine the key samples and frames to synchronize the corresponding haptic–visual signals. Without timestamps in the signal flow, the proposed method is still effective and more robust in complicated network conditions. Subjective evaluation also shows a significant improvement of user experience with the proposed method.

Conformable and Compact Multiaxis Tactile Sensor for Human and Robotic Grasping via Anisotropic Waveguides

AbstractRich and accurate tactile perceptive capability is important for both humans and robots. Soft materials exhibit unique characteristics to construct high‐performance tactile sensors such as high sensitivity and high resistance to overload. In this work, a multiaxis tactile sensor based on a soft anisotropic waveguide that can distinguish normal force and shear force, which can greatly expand its potential uses in practice, is reported. First, the anisotropy of the waveguide sensor's response to vector forces is validated numerically and experimentally, and then two of those anisotropic units are embedded into one cladding with a crossed‐over layout, to form a multiaxis sensor. Then, a calibration algorithm is implemented on this sensor and the reconstruction of vector forces is achieved at an average accuracy of 28.0 mN for normal forces and 81.1 mN for shear forces, both with the sensing range of 1 N. Using this device, three demonstrations are shown to give outlooks of its potential application in human and robotic grasping tasks: a wearable tactile sensor for collecting human's haptic data during operation, a uniaxial gyroscope, and a robotic gripper's tactile sensor for assisting an unlocking task with a key. This work is a step toward more functional soft waveguide‐based force sensors.

Deep multi-model fusion network based real object tactile understanding from haptic data

Deep multi-model fusion network based real object tactile understanding from haptic data

Multi-modal self-adaptation during object recognition in an artificial cognitive system

The cognitive connection between the senses of touch and vision is probably the best-known case of multimodality. Recent discoveries suggest that the mapping between both senses is learned rather than innate. This evidence opens the door to a dynamic multimodality that allows individuals to adaptively develop within their environment. By mimicking this aspect of human learning, we propose a new multimodal mechanism that allows artificial cognitive systems (ACS) to quickly adapt to unforeseen perceptual anomalies generated by the environment or by the system itself. In this context, visual recognition systems have advanced remarkably in recent years thanks to the creation of large-scale datasets together with the advent of deep learning algorithms. However, this has not been the case for the haptic modality, where the lack of two-handed dexterous datasets has limited the ability of learning systems to process the tactile information of human object exploration. This data imbalance hinders the creation of synchronized datasets that would enable the development of multimodality in ACS during object exploration. In this work, we use a multimodal dataset recently generated from tactile sensors placed on a collection of objects that capture haptic data from human manipulation, together with the corresponding visual counterpart. Using this data, we create a multimodal learning transfer mechanism capable of both detecting sudden and permanent anomalies in the visual channel and maintaining visual object recognition performance by retraining the visual mode for a few minutes using haptic information. Our proposal for perceptual awareness and self-adaptation is of noteworthy relevance as can be applied by any system that satisfies two very generic conditions: it can classify each mode independently and is provided with a synchronized multimodal data set.

Looking for haptics. Touch digitalization business strategies in luxury and fashion during COVID-19 and beyond

The sense of touch is central to fashion and luxury, because dress - garments and accessories - is experienced with and on the body. But on e-commerce websites, dress is dematerialized, and touch sensation simulated using audio-visual means. The growth of digital fashion communication during the COVID-19 pandemic has spotlighted this issue. However, developments in haptic technologies suggest that new modes of tactile interaction could be available in the online domain. Through interviews with fourteen fashion and luxury executives in Italy and Switzerland, this article investigates touch digitalization strategies during COVID-19 and the potential which haptics may hold to fill the online tactile void, using two surface haptic technologies as prompts. By doing so, this research fills a gap in the study of digital fashion and haptics, to the benefit of the academic and practitioner communities. Results indicate that executives deployed a host of visualization tactics during lockdown, reinforcing their conviction that still and moving images furnish an acceptable surrogate for touch sensation in the online domain. In fact, participants questioned the value of mediated tactile feedback provided by surface haptic devices. However, these experts also concurred that the technology holds promise, envisioning future use of haptic technologies in digital business-to-business and end-consumer markets. These could include haptic data integration within virtual prototyping, haptically enriched digital content for customer engagement, multisensory brand promotions in mixed or virtual reality, and phygital experiences.

Smart Manufacturing and Tactile Internet Based on 5G in Industry 4.0: Challenges, Applications and New Trends

For many applications deployed in manufacturing networks, communication latency has been a significant barrier. Despite the constant development of improved communication protocols and standards during Industry 4.0, the latency problem persists, lowering quality of services (QoS) and quality of experience (QoE). Tactile internet (TI), with its high availability, security, and ultra-low latency, will add a new dimension to human-machine interaction (HMI) by enabling haptic and tactile sensations. The tactile internet (TI) is a cutting-edge technology that uses 5G and beyond (B5G) communications to enable real-time interaction of haptic data over the internet between tactile ends. This emerging TI technology is regarded as the next evolutionary step for the Internet of Things (IoT) and is expected to bring about massive changes towards Society 5.0 and to address complex issues in current society. To that end, the 5G mobile communication systems will support the TI at the wireless edge. As a result, TI can be used as a backbone for delay mitigation in conjunction with 5G networks, allowing for ultra-reliable low latency applications like Smart Manufacturing, virtual reality, and augmented reality. Consequently, the purpose of this paper is to present the current state of 5G and TI, as well as the challenges and future trends for 5G networks beyond 2021, as well as a conceptual framework for integrating 5G and TI into existing industrial case studies, with a focus on the design aspects and layers of TI, such as the master, network, and slave layers. Finally, the key publications focused on the key enabling technologies of TI are summarized and the beyond 5G era towards Society 5.0 based on cyber-physical systems is discussed.

IMMERSIVE TELECOMMUNICATIONS: A SURVEY AND FUTURE DEVELOPMENT

Currently there is a growing need for technology to enhance the ability of people to interact remotely. The technologies available at the moment can always meet the needs of users, in particular in matters of providing telepresence, as well as the transmission of haptic data. The article proposes the introduction of a new term – immersive technologies in telecommunications, as well as an overview of technologies and future research related to this subject. Research subject is immersive telecommunication technologies. Research method is an analysis of the literature on the subject. Core results are analysis and survey of the immerse communications equipment and network requirements. Practical relevance of the work is that the results can be used for immerse communications applications design and development.

A Novel Energy Compensation Scheme for Quality Enhancement in Time-Delayed Teleoperation With Multi-DoF Haptic Data Reduction and Communication.

Efficient and low-delay exchange of hapticinformation plays a key role for the design of high-fidelity teleoperation systems that operate over real-world communication networks. In the presence of communication unreliabilities such as delay and packet loss, a combination of stability-ensuring control schemes and haptic data reduction approaches is essential to address the challenges from both the control and communication perspective. In this paper, we extend our previous one degree-of-freedom (DoF) solution to 3DoF that combines the time-domain passivity approach (TDPA) and the perceptual deadband-based (DB) haptic data reduction approach. We also extend the 1DoF energy compensation (EC) scheme to 3DoF for mitigating the control artifacts introduced by the inter-play of the TDPA and DB approaches. The 3DoF solution is analyzed in detail and verified through a real-world teleoperation setup. The performance of the proposed method and the existing solutions is compared though objective and subjective assessments. Experimental results show that by jointly considering the evaluated objective/subjective quality and the required packet rate, the proposed 3DoF solution with the EC scheme improves system performance significantly compared to the state-of-the-art solutions.

A Survey of the Tactile Internet: Design Issues and Challenges, Applications, and Future Directions

The Tactile Internet (TI) is an emerging area of research involving 5G and beyond (B5G) communications to enable real-time interaction of haptic data over the Internet between tactile ends, with audio-visual data as feedback. This emerging TI technology is viewed as the next evolutionary step for the Internet of Things (IoT) and is expected to bring about a massive change in Healthcare 4.0, Industry 4.0 and autonomous vehicles to resolve complicated issues in modern society. This vision of TI makes a dream into a reality. This article aims to provide a comprehensive survey of TI, focussing on design architecture, key application areas, potential enabling technologies, current issues, and challenges to realise it. To illustrate the novelty of our work, we present a brainstorming mind-map of all the topics discussed in this article. We emphasise the design aspects of the TI and discuss the three main sections of the TI, i.e., master, network, and slave sections, with a focus on the proposed application-centric design architecture. With the help of the proposed illustrative diagrams of use cases, we discuss and tabulate the possible applications of the TI with a 5G framework and its requirements. Then, we extensively address the currently identified issues and challenges with promising potential enablers of the TI. Moreover, a comprehensive review focussing on related articles on enabling technologies is explored, including Fifth Generation (5G), Software-Defined Networking (SDN), Network Function Virtualisation (NFV), Cloud/Edge/Fog Computing, Multiple Access, and Network Coding. Finally, we conclude the survey with several research issues that are open for further investigation. Thus, the survey provides insights into the TI that can help network researchers and engineers to contribute further towards developing the next-generation Internet.

Haptic Data Prediction and Extrapolation for Communication Traffic Reduction of Four-Channel Bilateral Control System

Robotic teleoperation with a bilateral control system has attracted attention owing to its haptic transmission performance. However, conventional bilateral control systems require broadband communication to transmit the vivid haptic sensation. This problem limits the application range of the bilateral control systems. The communication traffic can be reduced by predicting and extrapolating the incoming data. However, in the conventional prediction-based methods, only one type (e.g., position, velocity, or force) of data is transmitted per one direction because of the difficulty of predicting multiple independent data. The novelty of this article is the realization of the prediction-based traffic reduction in the four-channel bilateral control system that transmits accurate haptic sensation by communicating both position and force data. By equivalently transforming this control scheme in the structure of impedance control, the transmit data are summed up to one data, equilibrium force. The equilibrium force is not only transmitted but extrapolated on the receiver side. As a result, the communication frequency becomes low without degrading haptic transmission performance. The validity of the proposed method was confirmed through experiments and succeeded to reduce the communication data size to less than 3.0%. The proposed method helps to realize a high-performance bilateral control system on band-limited networks.

A Network-Adaptive Prediction Algorithm for Haptic Data Under Network Impairments

Real-time tele-haptic applications require capturing, compressing, transmitting, and displaying haptic information, which includes tactile and kinesthetic information. To achieve a high quality of service (QoS), real-time haptic data stream synchronization between local and remote environments is required. However, transmission of data over a computer network is often affected by network impairments, such as network delay, jitter, and packet loss, thus leading to system instability and poor performance. Current prediction algorithms for networked haptics comprise perceptual data reduction, traffic prioritization approaches, congestion control approaches, and radio resource allocation. However, the mentioned prediction algorithms either do not consider packet loss and time-varying delays (i.e., jitter) in their experimental setup, or only consider packet loss or delays. In real-world network environments, both packet loss and delays often occur simultaneously. In this work, a network adaptive Trust Strategy Prediction (TSP) algorithm was modified to work under both network impairments. The objective of the TSP is to maintain real-time haptic synchronization (haptic data stream synchronization) between the haptic interactive environments, by compensating network impairments using selective and specific prediction strategies, according to changes in the network’s characteristics. The experimental results demonstrate that TSP offers greater accuracy and smaller inconsistencies in terms of the predicted position, compared to the dead reckoning prediction and velocity estimation, which is often employed with filtering techniques.

Edge-Learning-Enabled Realistic Touch and Stable Communication for Remote Haptic Display

As the basis of Tactile Internet, remote haptic display has been made possible with the development of ultra-reliable low-latency communication in 5G. In this study, edge learning is employed to enable realistic haptic display and stable remote communication. We propose a double-loop control algorithm, which merges decoupling and PID neural network, for magnetic field generation of the electromagnetic haptic device. In addition, a supervised bidirectional LSTM network is constructed for online haptic prediction during remote interaction, thus complementing the missing haptic data on account of time delay and packet loss in network communications. Experiments have been conducted on the built remote haptic display system, where data streams from sensors are gathered, stored, and forwarded in real time. The results show that dynamic and accurate haptic display is achieved through our magnetic field control algorithm for the haptic device, and the error of haptic prediction by step is less than 0.01N. The conclusion is that the service sustainability of remote haptic display can be guaranteed by edge learning effectively.