Interaction Modalities Research Articles

Interpreting technologized

Abstract As an introduction to a selection of current research on the theme of interpreting and technology this article offers an overview of the technologization of interpreting in the course of the past century, with particular emphasis on developments during recent decades. The application of various types of technology in different professional domains and settings is discussed with reference to the main functions of technology: enabling service delivery, assisting the process and replacing human agency. A brief review of some (proto)typical applications of technology in different settings and modalities of interaction then serves as a backdrop to a summary discussion of the technology-related literature to date and an introduction to the current research presented in Interpreting 26:2 (2024).

Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework

AbstractDuring aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three input modes: voice inputs, situation inputs, and plan inputs. This research focuses on the design of an aircraft human‐machine interaction assistant (AHMIA), which serves as a multimodal data processing and application framework for human‐to‐machine interaction in a fully voice‐controlled manner. For the voice mode, a finetuned FunASR model is employed, leveraging private aeronautical datasets to enable specific aeronautical speech recognition. For the situation mode, a hierarchical situation events extraction model is proposed, facilitating the utilization of high‐level situational features. For the plan mode, a multi‐formations double‐code network plan diagram with a timeline is utilized to effectively represent plan information. Notably, to bridge the gap between human language and machine language, a hierarchical knowledge engine named process‐event‐condition‐order‐skill (PECOS) is introduced. PECOS provides three distinct products: the PECOS model, the PECOS state chart, and the PECOS knowledge description. Simulation results within the air confrontation scenario demonstrate that AHMIA enables active‐command and passive‐response interactions with pilots, thereby enhancing the overall interaction modality.

An integrated prototyping tool to enhance interdisciplinary communication in smart product design

The increasing integration of smart products into society is noteworthy in the field of Prospective Ergonomics (PE), as they herald the emergence of novel products featuring yet-to-be-discovered interaction modalities. Literature highlights a lack of front-end prototyping approaches that transcend disciplinary skills and aid communication across diverse design fields, highly relevant in early-stage smart product development. This paper introduces an integrated, low-fidelity prototyping tool to enhance interdisciplinary communication and understanding of future products’ interactive and environmental qualities. A survey provides preliminary validation of the integrated tool’s format and content. Findings endorse the tool’s structure and flexibility in supporting understanding and communication of interactive and environmental qualities of smart product concepts across a range of design disciplines. As such we advance the discourse on PE by envisioning a low-fidelity prototyping tool to create intermediary objects supporting design efforts towards future products and their new and emerging interaction modalities.

Enhancing clinical reasoning skills for medical students: a qualitative comparison of LLM-powered social robotic versus computer-based virtual patients within rheumatology.

Virtual patients (VPs) are increasingly used in medical education to train clinical reasoning (CR) skills. However, optimal VP design for enhancing interactivity and authenticity remains unclear. Novel interactive modalities, such as large language model (LLM)-enhanced social robotic VPs might increase interactivity and authenticity in CR skill practice. To evaluate medical students' perceptions of CR training using an LLM-enhanced social robotic VP platform compared with a conventional computer-based VP platform. A qualitative study involved 23 third-year medical students from Karolinska Institutet, who completed VP cases on an LLM-enhanced social robotic platform and a computer-based semi-linear platform. In-depth interviews assessed students' self-perceived acquirement of CR skills using the two platforms. Thematic analysis was employed to identify themes and sub-themes. Three main themes were identified: authenticity, VP application, and strengths and limitations. Students found the social robotic platform more authentic and engaging. It enabled highly interactive communication and expressed emotions, collectively offering a realistic experience. It facilitated active learning, hypothesis generation, and adaptive thinking. Limitations included lack of physical examination options and, occasionally, mechanical dialogue. The LLM-enhanced social robotic VP platform offers a more authentic and interactive learning experience compared to the conventional computer-based platform. Despite some limitations, it shows promise in training CR skills, communication, and adaptive thinking. Social robotic VPs may prove useful and safe learning environments for exposing medical students to diverse, highly interactive patient simulations.

A shared robot control system combining augmented reality and motor imagery brain–computer interfaces with eye tracking

Objective: Brain-computer interface (BCI) control systems monitor neural activity to detect the user's intentions, enabling device control through mental imagery. Despite their potential, decoding neural activity in real-world conditions poses significant challenges, making BCIs currently impractical compared to traditional interaction methods. This study introduces a novel motor imagery (MI) BCI control strategy for operating a physically assistive robotic arm, addressing the difficulties of MI decoding from electroencephalogram (EEG) signals, which are inherently non-stationary and vary across individuals.
Approach: A proof-of-concept BCI control system was developed using commercially available hardware, integrating MI with eye tracking in an augmented reality (AR) user interface to facilitate a shared control approach. This system proposes actions based on the user's gaze, enabling selection through imagined movements. A user study was conducted to evaluate the system's usability, focusing on its effectiveness and efficiency.
Main results:Participants performed tasks that simulated everyday activities with the robotic arm, demonstrating the shared control system's feasibility and practicality in real-world scenarios. Despite low online decoding performance (mean accuracy: 0.52 9, F1: 0.29, Cohen's Kappa: 0.12), participants achieved a mean success rate of 0.83 in the final phase of the user study when given 15 minutes to complete the evaluation tasks. The success rate dropped below 0.5 when a 5-minute cutoff time was selected.
Significance: These results indicate that integrating AR and eye tracking can significantly enhance the usability of BCI systems, despite the complexities of MI-EEG decoding. While efficiency is still low, the effectiveness of our approach was verified. This suggests that BCI systems have the potential to become a viable interaction modality for everyday applications
in the future.

Synthesis, Docking Study, and Antimicrobial Activity of Some New Heterocyclic Compounds Bearing Naphthalene Moiety

AbstractUpon reaction with α,β‐unsaturated nitriles (2 a–d) and/or salicylaldehyde, 3‐(naphthalen‐2‐yl)‐3‐oxopropanenitrile (1) yields pyridine derivatives 3 a–d and/or the imino chromene derivative 4. Compound 1 reacted with phenyl isothiocyanate in a basic medium to give the intermediate potassium salt 6. Upon stirring at room temperature, the intermediate 6 with some α‐halogenated compounds yielded the acyclic compounds 7, 9, 11, 13, and 15. On the other hand, refluxing intermediate 6 with the same α‐halogenated compounds in DMF in the presence of TEA afforded the corresponding thiophene derivatives 8, 10, 12, 14, and 16. The reaction of thiol 17 with hydrazine hydrate derivatives 18 a–d yielded the corresponding pyrazole derivatives 21 a–d. Compound 20 reacted with o‐phenylenediamine or o‐aminothiophenol in the presence of DMF/TEA to give the corresponding benzothiazole or benzoimidazole derivatives 27 a and 27 b, respectively. Thirty new synthesized compounds were examined for their antibacterial activity against Gram‐positive bacteria including S. aureus and B. subtilis, Gram‐negative bacteria such as P. aeruginosa and E. coli, and yeast‐like fungi like C. albicans in vitro. Compounds 15, 16, 21 d, 27 a, 27 b, exhibited the highest antibacterial activity against Gram‐positive bacteria. Molecular docking experiments were conducted to explore the binding affinities and interaction modalities of selected derivatives with the target PDB‐3cmt protein.

Assessing Creativity and User Experience in Immersive Virtual Reality with Cultural Heritage Learning

Virtual reality (VR) technology is widely used in the field of education and cultural heritage learning. However, few studies have focused on the impact on creativity regarding the use of different interaction modalities by users in cultural heritage learning. Furthermore, it is controversial whether VR can enhance user experience and learning outcomes. The study utilized a mixed research approach combining qualitative and quantitative methods, where qualitative data was collected using open-ended interviews, simultaneously quantitative data was collected using learning outcomes, intrinsic motivation, workload, self-report of creativity and expert assessments of creativity. A mixed research approach combining qualitative and quantitative methods was used; creativity self-reports were used to collect qualitative data, and learning outcomes, intrinsic motivation, and workload data were collected for the quantitative findings.90 university students were randomly assigned to the text group (TG), the video group (VG), and the immersive virtual reality group (IVRG), with 30 students in each group. The study assessed whether creativity, user experience and learning outcomes differed when using different interaction modalities in a cultural heritage context. The results demonstrated that the text interaction modality outperformed the other two groups in learning outcomes and expert assessments of creativity performance, while the IVRG performed better in self-report of creativity and user experience. In addition, participants in TG had higher workloads and poorer user experiences compared to the other two interaction modalities. This study encourages that different interaction modalities should be selected more appropriately depending on the purpose. The study found that interaction modalities creativity can be applied to digital cultural heritage and education and informs interaction design for the HCI community.

ScentClue: Enhancing Story Engagement in Virtual Reality Through Hedonically Varied Olfactory Hints

To enhance the engagement and significance of olfaction as a novel medium in multimedia systems, we aim to integrate different hedonic odorous cues into narratives to augment immersion in virtual experiences. We introduce the ScentClue system, comprising (1) an original and murder-themed experimental film incorporating four distinct hedonic odor cues synchronized with audiovisual elements, (2) a neck-worn scent delivery device capable of dispensing four different odors, and (3) an Unreal Engine-based virtual cinema environment enabling rapid control of video playback and odor dissemination. Experimenting with 18 participants divided into control groups with and without odor stimuli, we validated the system’s usability and effectiveness of odor cue settings. Furthermore, we found that the use of odors with different hedonic qualities could enhance odor recognition, amplify thematic story ambiance, and aid in the rapid differentiation of complex relationships. Finally, we discussed expanded scenarios and interaction modalities for ScentClue, proposing future narrative-driven olfaction-enhanced multimedia designs with considerations for odor cue settings, odor material selection, and scent-releasing setup, particularly unpleasant odors.

Indazole to 2-Cyanoindole Scaffold Progression for Mycobacterial Lipoamide Dehydrogenase Inhibitors Achieves Extended Target Residence Time and Improved Antibacterial Activity.

Tuberculosis remains a leading cause of death from a single infection worldwide. Drug resistance to existing and even new antimycobacterials calls for research into novel targets and unexplored mechanisms of action. Recently we reported on the development of tight-binding inhibitors of Mycobacterium tuberculosis (Mtb) lipoamide dehydrogenase (Lpd), which selectively inhibit the bacterial but not the human enzyme based on a differential modality of inhibitor interaction with these targets. Here we report on the striking improvement in inhibitor residence time on the Mtb enzyme associated with scaffold progression from an indazole to 2-cyanoindole. Cryo-EM of Lpd with the bound 2-cyanoindole inhibitor 19 confirmed displacement of the buried water molecule deep in the binding channel with a cyano group. The ensuing hours-long improvement in on-target residence time is associated with enhanced antibacterial activity in axenic culture and in primary mouse macrophages. Resistance to 2-cyanoindole inhibitors involves mutations within the inhibitor binding site that have little effect on inhibitor affinity but change the modality of inhibitor-target interaction, resulting in fast dissociation from Lpd. These findings underscore that on-target residence time is a major determinant of antibacterial activity and in vivo efficacy.

Indazole to 2‐Cyanoindole Scaffold Progression for Mycobacterial Lipoamide Dehydrogenase Inhibitors Achieves Extended Target Residence Time and Improved Antibacterial Activity

AbstractTuberculosis remains a leading cause of death from a single infection worldwide. Drug resistance to existing and even new antimycobacterials calls for research into novel targets and unexplored mechanisms of action. Recently we reported on the development of tight‐binding inhibitors of Mycobacterium tuberculosis (Mtb) lipoamide dehydrogenase (Lpd), which selectively inhibit the bacterial but not the human enzyme based on a differential modality of inhibitor interaction with these targets. Here we report on the striking improvement in inhibitor residence time on the Mtb enzyme associated with scaffold progression from an indazole to 2‐cyanoindole. Cryo‐EM of Lpd with the bound 2‐cyanoindole inhibitor 19 confirmed displacement of the buried water molecule deep in the binding channel with a cyano group. The ensuing hours‐long improvement in on‐target residence time is associated with enhanced antibacterial activity in axenic culture and in primary mouse macrophages. Resistance to 2‐cyanoindole inhibitors involves mutations within the inhibitor binding site that have little effect on inhibitor affinity but change the modality of inhibitor‐target interaction, resulting in fast dissociation from Lpd. These findings underscore that on‐target residence time is a major determinant of antibacterial activity and in vivo efficacy.

Effects of Self-Learning and Exploration for XR-based Interactions

This research explores the overtime learning trends of multimodal gaze-based interactions in tasks involving the movement of augmented objects within extended reality (XR) environments. This study employs three interactions, including two multimodal gaze-based approaches, and compares them with an unimodal hand-based interaction. The underlying hypothesis posits that gaze-based interactions outperform other modalities, promising improved performance, lower learnability rates, and enhanced efficiency. These assertions serve as the foundation for investigating the dynamics of self-learning and exploration within XR-based environments. To this end, the study addresses questions related to the temporal evolution of learnability, post-learning efficiency, and users’ subjective preferences regarding these interaction modalities. This research shows that gaze-based interactions enhance performance, exhibit a lower learnability rate, and demonstrate higher efficiency compared to an unimodal hand-based interaction. Our results contribute to the design and refinement of more effective, user-friendly, and adaptive XR user interfaces.

Exploring the effects of multiple online interaction on emotions of L2 learners in synchronous online classes

The decline in both the quantity and quality of interaction has emerged as a notable challenge in online learning. However, the definition of interaction quality remains unclear. This study clarifies it as a decrease in the breadth of interaction, which refers to interaction that can only cover a smaller number of learners. To address this, a synchronous interaction modality, termed Multiple Online Interaction (MOI), based on Zoom's interactive tools, was introduced. In a quasi-experiment involving 58 Chinese L2 learners (with 30 beginner and 28 intermediate students), emotions were assessed using the Brief Mood Introspection Scale (BMIS) while real-time emotional dynamics were revealed through the analysis of 5129 facial expression images during a 35-min synchronous class. MOI participants reported higher levels of Lively and Happy but also experienced more Nervous and less Calm. These emotional dynamics, tracked through expression recognition technology, demonstrate that MOI's impact is primarily observed during the first Grammar & Practice section of the teaching. The empirical findings of this study provide practical insights for educators aiming to conduct effective online teaching in the future.

Interactive Table Synthesis With Natural Language.

Tables are a ubiquitous data format for insight communication. However, transforming data into consumable tabular views remains a challenging and time-consuming task. To lower the barrier of such a task, research efforts have been devoted to developing interactive approaches for data transformation, but many approaches still presume that their users have considerable knowledge of various data transformation concepts and functions. In this study, we leverage natural language (NL) as the primary interaction modality to improve the accessibility of average users to performing complex data transformation and facilitate intuitive table generation and editing. Designing an NL-driven data transformation approach introduces two challenges: 1) NL-driven synthesis of interpretable pipelines and 2) incremental refinement of synthesized tables. To address these challenges, we present NL2Rigel, an interactive tool that assists users in synthesizing and improving tables from semi-structured text with NL instructions. Based on a large language model and prompting techniques, NL2Rigel can interpret the given NL instructions into a table synthesis pipeline corresponding to Rigel specifications, a declarative language for tabular data transformation. An intuitive interface is designed to visualize the synthesis pipeline and the generated tables, helping users understand the transformation process and refine the results efficiently with targeted NL instructions. The comprehensiveness of NL2Rigel is demonstrated with an example gallery, and we further confirmed NL2Rigel's usability with a comparative user study by showing that the task completion time with NL2Rigel is significantly shorter than that with the original version of Rigel with comparable completion rates.

Employer–Domestic Worker Dynamic in Kolkata: From Maternalism to Transactionalism

Domestic workers used to be treated virtually as members of the family in South Asian contexts, with much evidence of various forms of exploitation. Presently, the increasing prevalence of formal patron–client dynamics among part-time domestic labourers in urban India takes specific hybrid forms that require further research. The article examines to what extent efforts to establish purely commercial arrangements in the domestic service sector may avoid traditional forms of exploitation. Or do more formal professional relationships still tend to benefit employers more, allowing them to assign additional tasks without offering corresponding extra benefits? The findings identify that establishing and maintaining purely professional relationships remains challenging in India due to deep-rooted ethical and cultural considerations and adaptability issues. Employers often unwittingly perpetuate traditional personalised domestic work settings. The article shows how both sides in this complex relationship struggle to identify and practice new modalities of fair interaction.

An affordable, field-deployable detecting system for cyanide ion – Investigating applications in real time uses

A highly efficient system that incorporates the instantaneous visualization of the cyanide ion in water was synthesized by keeping the fluorophore system (electron donor) as a julolidine-coumarin conjugate and changing the electron acceptor unit. The probes exhibit a notable color change in normal and UV light. The probe interaction modalities are based on the ICT process. With a detection limit in the nM range, it may preferentially react with cyanide, which is less than the tolerable level of 1.9 μM. According to 1H NMR data, the probes detect cyanide ions by nucleophilic addition reaction mechanism. Furthermore, current probe successfully determines real resources, including cyanide containing cassava powder, sprouted potatoes and various water samples. Besides the test strips, an electronic Arduino device was also employed to detect the cyanide ion. As such, the developed probes exhibit outstanding practical application with respect to the cyanide ion.

Comparing online wrist and forearm EMG-based control using a rhythm game-inspired evaluation environment

Objective.The use of electromyogram (EMG) signals recorded from the wrist is emerging as a desirable input modality for human-machine interaction (HMI). Although forearm-based EMG has been used for decades in prosthetics, there has been comparatively little prior work evaluating the performance of wrist-based control, especially in online, user-in-the-loop studies. Furthermore, despite different motivating use cases for wrist-based control, research has mostly adopted legacy prosthesis control evaluation frameworks.Approach.Gaining inspiration from rhythm games and the Schmidt's law speed-accuracy tradeoff, this work proposes a new temporally constrained evaluation environment with a linearly increasing difficulty to compare the online usability of wrist and forearm EMG. Compared to the more commonly used Fitts' Law-style testing, the proposed environment may offer different insights for emerging use cases of EMG as it decouples the machine learning algorithm's performance from proportional control, is easily generalizable to different gesture sets, and enables the extraction of a wide set of usability metrics that describe a users ability to successfully accomplish a task at a certain time with different levels of induced stress.Main results.The results suggest that wrist EMG-based control is comparable to that of forearm EMG when using traditional prosthesis control gestures and can even be better when using fine finger gestures. Additionally, the results suggest that as the difficulty of the environment increased, the online metrics and their correlation to the offline metrics decreased, highlighting the importance of evaluating myoelectric control in real-time evaluations over a range of difficulties.Significance.This work provides valuable insights into the future design and evaluation of myoelectric control systems for emerging HMI applications.

Effect of Human Interaction Modalities on the Usability and Performance of a Robotic Cyber-Physical System for Construction Progress Monitoring

The construction industry has been slow to adopt advanced technologies like Cyber-Physical Systems (CPS), underscoring the necessity for a human-centered approach to their development. This study aims to understand the significance and impact of human-CPS interaction modalities on human perceptions of CPS, specifically in the use case of construction progress monitoring. A mixed-method approach comprising quantitative observations and qualitative interviews was employed to evaluate the system’s usability and performance with different human-CPS interaction modalities. The findings reveal variable levels of usability and performance across different human-CPS interaction modalities. Users expressed a preference for the three-dimensional immersive environment compared to a two-dimensional setting due to a better sense of presence. However, they experienced heightened cognitive load when utilizing novel interaction methods such as hand gestures and voice commands. Interestingly, despite this increased cognitive demand, users exhibited comparable performance in the assigned task, irrespective of their familiarity with the interaction modality. The results underscore the importance of human-centered design in CPS development. It contributes empirical evidence crucial for understanding and improving human-CPS interactions, thereby advancing the implementation of CPS in construction progress monitoring.

Exploring the effect of human-drone communication modality on safety and balance control in virtual construction environments

This study examines the impact of Human-Drone Interaction (HDI) modalities on construction workers’ safety and balance control within virtual environments. Utilising virtual reality (VR) simulations, the study explored how gesture and speech-based communications influence workers’ physical postures and balance, contrasting these modalities with a non-interactive control group. One hundred participants were recruited, and their movements and balance control were tracked using motion sensors while they interacted with virtual drones through either gesture, speech, or without communication. Results showed that interactive modalities significantly improved balance control and reduced the risk of falls, suggesting that advanced HDI can enhance safety on construction sites. However, speech-based interaction increased cognitive workload, highlighting a trade-off between physical safety and mental strain. These findings underscore the potential of integrating intuitive communication methods into construction operations, although further research is needed to optimise these interactions for long-term use and in diverse noise environments.

Modifications to consent documentation with adults with communication disorders following brain injury: An exploratory study.

Consent documentation for research studies is often inaccessible to people with neurogenic communication disorders following brain injury and there is limited literature on specific modifications for informed consent. This exploratory study aimed to identify effective strategies and modifications to consent processes for adults with brain injury. Using a fictitious research study, we developed a set of Participant Information Sheets (PISs) varying in complexity, presentation format, and communication modality. Evaluations were conducted with eight participants. Findings indicated diverse participant preferences for PIS modifications, suggesting simplified vocabulary, reduced text, carefully selected images, and an interactive presentation modality as helpful strategies. Building on previous literature, we present refined guidelines for consent modifications for adults with neurogenic communication disorder after brain injury. These guidelines can promote more appropriate inclusion of communicatively impaired populations in research and assist ethics committees and researchers in preparing modified consent documents.

Competency-Based HR Management Strategy in the Digital Era: Systematic Literature Review

This research conducts a systematic literature review (SLR) to explore competency-based human resource management (HRM) strategies in the digital era. The analysis reveals that developing digital competencies ranks high on the agenda for many organizations. Continuous investment in training and development programs to enhance employees' digital skills is deemed crucial. Moreover, fostering a work environment conducive to creativity and flexibility is emphasized, where employees are empowered to innovate and swiftly adapt to technological advancements. The study also identifies generational disparities in the workplace as a determinant influencing training and development methodologies. Millennials, for instance, exhibit greater receptivity to technology- driven and interactive learning modalities, necessitating adaptable trainin g strategies that cater to diverse learning preferences. Integrating technology into HR management systems emerges as a pivotal step. Advanced HR management information systems facilitate efficient handling of employee data, streamline competency tracking, and pinpoint training requisites with greater precision. Technologies such as big data and analytics additionally furnish profound insights into employee performance and areas ripe for enhancement. In conclusion, this study underscores that competency-based HRM strategies in the digital era must be agile, innovative, and technology-enabled. Organizations proficient in deploying these strategies are better positioned to confront challenges and harness opportunities arising from digital transformation. The findings offer crucial insights for HR management practitioners to craft and implement effective HR development strategies tailored for the digital age, aligning with Sustainable Development Goals (SDGs) like SDG 8 (Decent Work and Economic Growth) and SDG 9 (Industry, Innovation, and Infrastructure).