Human-machine Collaboration Research Articles

Teachable Conversational Agents for Crowdwork: Effects on Performance and Trust

Traditional crowdsourcing has mostly been viewed as requester-worker interaction where requesters publish tasks to solicit input from human crowdworkers. While most of this research area is catered towards the interest of requesters, we view this workflow as a teacher-learner interaction scenario where one or more human-teachers solve Human Intelligence Tasks to train machine learners. In this work, we explore how teachable machine learners can impact their human-teachers, and whether they form a trustable relation that can be relied upon for task delegation in the context of crowdsourcing. Specifically, we focus our work on teachable agents that learn to classify news articles while also guiding the teaching process through conversational interventions. In a two-part study, where several crowd workers individually teach the agent, we investigate whether this learning by teaching approach benefits human-machine collaboration, and whether it leads to trustworthy AI agents that crowd workers would delegate tasks to. Results demonstrate the benefits of the learning by teaching approach, in terms of perceived usefulness for crowdworkers, and the dynamics of trust built through the teacher-learner interaction.

Partner or subordinate? Sequential risky decision-making behaviors under human-machine collaboration contexts

Human-machine teaming has shown great potential in sequential risky decision-making (SRDM), and it is promising that machines will no longer work as subordinates. Technical advances prompt people to consider the contexts of humans and machines sharing decision authority. This study aims to compare task performance, human behaviors, and subjective perception of machines in a human-dominated team (machine as a subordinate) versus a human-machine joint team (machine as a partner). We modified the Balloon Analogue Risk Task (BART) experiment to include a highly accurate machine and accommodate two types of human-machine teams (HMTs). The results showed that both HMTs yielded comparable task performance and overperformed human or machine deciding alone. In the human-machine joint team, the machine as a partner entailed human decision-makers to cede power and coordinate, and their pumping decisions became more conservative and fluctuating. Moreover, human decision-makers reacted more sensitively to the different results of the last trial. Although subjects generally favored working with a machine subordinate, they exhibited similar trust levels in both HMTs after sufficient interaction. Our preliminary findings show that allocating a partial decision authority to highly accurate machines changed human behaviors without impairing task performance or trust in collaboration.

Computer vision-based human-machine collision warning system on construction site

With the increase of mechanization in the construction industry, more and more construction machines appear on construction sites. This improves construction efficiency but leads to the high risk of human-machine collision. At present, human-machine collision risks are usually monitored and warned manually. It is difficult to identify relevant risks in time due to the lack of sufficient safety supervisors. This paper proposes an efficient human-machine collision warning system based on computer vision and deep learning. The designed system and relevant devices are installed on construction machinery. It can quickly identify workers and measure the distance between workers and machinery in the complex and dynamic environment so as to quickly warn the risk of human-machine collision based on a threshold. Power supply restriction and harsh environment such as dust and vibration are also considered to ensure the stable and effective operation of the system. The system was tested on a construction site for 6 weeks. The result shows that the system has a significant positive effect on the early warning of human-machine collision risk, thus benefiting the improvement of safety performance of workers in human-machine collaboration scenes.

An artificial remote tactile device with 3D depth-of-field sensation.

Flexible tactile neuromorphic devices are becoming important as the impetus for the development of human-machine collaboration. However, accomplishing and further transcending human intelligence with artificial intelligence still confront many barriers. Here, we present a self-powered stretchable three-dimensional remote tactile device (3D-RTD) that performs the depth-of-field (DOF) sensation of external mechanical motions through a conductive-dielectric heterogeneous structure. The device can build a logic relationship precisely between DOF motions of an external active object and sensory potential signals of bipolar sign, frequency, amplitude, etc. The sensory mechanism is revealed on the basis of the electrostatic theory and multiphysics modeling, and the performance is verified via an artificial-biological hybrid system with micro/macroscale interaction. The feasibility of the 3D-RTD as an obstacle-avoidance patch for the blind is systematically demonstrated with a rat. This work paves the way for multimodal neuromorphic device that transcends the function of a biological one toward a new modality for brain-like intelligence.

The Next Frontier in Intelligent Augmentation: Human-Machine Collaboration in Strategic Marketing Decision-Making

Sciendo provides publishing services and solutions to academic and professional organizations and individual authors. We publish journals, books, conference proceedings and a variety of other publications.

A Survey on Medical Explainable AI (XAI): Recent Progress, Explainability Approach, Human Interaction and Scoring System.

The emerging field of eXplainable AI (XAI) in the medical domain is considered to be of utmost importance. Meanwhile, incorporating explanations in the medical domain with respect to legal and ethical AI is necessary to understand detailed decisions, results, and current status of the patient’s conditions. Successively, we will be presenting a detailed survey for the medical XAI with the model enhancements, evaluation methods, significant overview of case studies with open box architecture, medical open datasets, and future improvements. Potential differences in AI and XAI methods are provided with the recent XAI methods stated as (i) local and global methods for preprocessing, (ii) knowledge base and distillation algorithms, and (iii) interpretable machine learning. XAI characteristics details with future healthcare explainability is included prominently, whereas the pre-requisite provides insights for the brainstorming sessions before beginning a medical XAI project. Practical case study determines the recent XAI progress leading to the advance developments within the medical field. Ultimately, this survey proposes critical ideas surrounding a user-in-the-loop approach, with an emphasis on human–machine collaboration, to better produce explainable solutions. The surrounding details of the XAI feedback system for human rating-based evaluation provides intelligible insights into a constructive method to produce human enforced explanation feedback. For a long time, XAI limitations of the ratings, scores and grading are present. Therefore, a novel XAI recommendation system and XAI scoring system are designed and approached from this work. Additionally, this paper encourages the importance of implementing explainable solutions into the high impact medical field.

Human-centric collaborative assembly system for large-scale space deployable mechanism driven by Digital Twins and wearable AR devices

The mass personalization of spacecraft challenges the experience and skill of assembly operators, which is caused by the poor collaboration between the elements in assembly. A large number of research works show that the role of human is irreplaceable in the process of assembly, but few publications focus on improving the collaboration to reduce the strict requirement for operators. To address those challenges, this paper proposes a human-centric collaborative assembly system (HCAS) for Large-scale Space Deployable Mechanism (LSDM) that is the core supporting structure of satellite, to solve the problem of poor collaboration between human and the resources in assembly site. Three Digital Twins (DTs), i.e., assembly process DT, assembly operator DT, and assembly quality DT are developed to achieve the collaboration among the elements of human-cyber-physical in HCAS. The breakthroughs brought by the three DTs are 1) adaptive assembly process sensing based on real-virtual scenes fusion with the help of assembly process DT and wearable AR device enhances the synergy between physical system and cyber system; 2) Based on the assembly operator DT and the Knowledge Graph, the adaptive assembly tasks assignment with operator-centric is realized; 3) Dynamic tolerance allocation of LSDM assembly considering space service environment based on assembly quality DT guides the operator to adjust the process parameters in real time. In addition, in view of the special requirement of gravity compensation of LSDM assembly, the method of using Co-robot to simulate microgravity and AR to plan robot path online is included to improve the human-machine collaboration. Finally, a prototype system is developed and applied in assembly site. Application results show that the developed method and system reduce the dependence on human experience and debases the frequency of rework, which achieve the purpose of improving assembly efficiency and quality.

RATING: Medical knowledge-guided rheumatoid arthritis assessment from multimodal ultrasound images via deep learning

SummaryMultimodal ultrasound has demonstrated its power in the clinical assessment of rheumatoid arthritis (RA). However, for radiologists, it requires strong experience. In this paper, we propose a rheumatoid arthritis knowledge guided (RATING) system that automatically scores the RA activity and generates interpretable features to assist radiologists' decision-making based on deep learning. RATING leverages the complementary advantages of multimodal ultrasound images and solves the limited training data problem with self-supervised pretraining. RATING outperforms all of the existing methods, achieving an accuracy of 86.1% on a prospective test dataset and 85.0% on an external test dataset. A reader study demonstrates that the RATING system improves the average accuracy of 10 radiologists from 41.4% to 64.0%. As an assistive tool, not only can RATING indicate the possible lesions and enhance the diagnostic performance with multimodal ultrasound but it can also enlighten the road to human-machine collaboration in healthcare.

Dynamic Hand Gesture Recognition Using Electrical Impedance Tomography.

Electrical impedance tomography (EIT) has been applied in the field of human-computer interaction due to its advantages including the fact that it is non-invasive and has both low power consumption and a low cost. Previous work has focused on static gesture recognition based on EIT. Compared with static gestures, dynamic gestures are more informative and can achieve more functions in human-machine collaboration. In order to verify the feasibility of dynamic gesture recognition based on EIT, a traditional excitation drive pattern is optimized in this paper. The drive pattern of the fixed excitation electrode is tested for the first time to simplify the measurement process of the dynamic gesture. To improve the recognition accuracy of the dynamic gestures, a dual-channel feature extraction network combining a convolutional neural network (CNN) and gated recurrent unit (GRU), namely CG-SVM, is proposed. The new center distance loss is designed in order to simultaneously supervise the intra-class distance and inter-class distance. As a result, the discriminability of the confusing data is improved. With the new excitation drive pattern and classification network, the recognition accuracy of different interference data has increased by 2.7~14.2%. The new method has stronger robustness, and realizes the dynamic gesture recognition based on EIT for the first time.

Interaction force measurement of parallel robots based on structure-integrated force sensors using interpretable linear neural networks

The parallel robot in this research is mainly designed for the human-machine collaboration, which has two working modes: follow-up mode and pressing mode. In order to better realize these two functions, the measurement of the interaction force is mainly concerned. In the arrangement of the hardware, different from the six-dimensional force sensor method and current estimation method commonly used in industry, the structure-integrated force sensors (SIFS) measurement method is used. In the calculation method, a back propagation neural networks (BPNN) method is introduced to calculate the interaction force. Then, based on the prior knowledge of physics, a new interpretable linear neural networks (ILNN) method is proposed. The details of the ILNN algorithm and its differences from the classic neural network methods are given. Finally, the experimental results of the direct Newton–Euler method, the least square method, the BPNN method, and the ILNN method are compared. Experimental results show that the inference accuracy of the ILNN method is slightly better than the direct Newton–Euler method and the least square method, and its generalization ability is much better than the BPNN method. The use of the ILNN method based on the FIFS arrangement makes the measurement of the interaction force more accurate and reliable.

NOMA-based cognitive radio network with hybrid FD/HD relay in industry 5.0

The continued growth of the industrial Internet of Things (IoT) paves the way for the fifth industrial revolution (Industry 5.0), which is expected to further enhance the wireless connectivity and human machine collaboration. Nonetheless, Industry 5.0 will inevitably face the great challenges such as large scale connectivity and explosive data traffic because of the proliferation of IoT devices. To meet these challenges, we propose a novel non-orthogonal multiple access (NOMA)-based overlay cognitive radio (OCR) network for Industry 5.0. Specifically, the hybrid full-duplex (FD)/ half-duplex (HD) mode at the secondary user (SR) is adopted and in-phase and quadrature phase imbalance (IQI) is introduced to better comply with the practical scenarios. We investigate the outage probability (OP) and ergodic sum rate (ESR) of the considered network with the Nakagami-m channels. Extensive simulation experiments are performed to verify the accuracy of the analysis. From the simulation results, we can conclude: i) Increasing the signal-to-noise ratio (SNR), OP decreases but ESR increases and tends to a fixed value at high SNR regimes, indicating that the existence of IQI leads to a ceiling for the ESR; ii) The primary system has higher reliable performance at low SNR regions when SR is in FD mode, and for ergodicity, the system with FD relay is superior to HD relay; iii) The reliable performance of OMA system for primary users will be better since there is no interference at the relay.

Human–machine collaboration in transcription

As automatic speech recognition (ASR) has improved, it has become a viable tool for content transcription. Prior to the use of ASR for this task, content transcription was achieved through human effort alone. Despite improvements, ASR performance is as yet imperfect, especially in more challenging conditions (eg multiple speakers, noise, nonstandard accents). Given this, a promising way forward is a human-in-the-loop (HIL) approach. This contribution describes our work with HIL ASR on the transcription task. Traditionally, ASR performance has been measured using word error rate (WER). This measure may not be sufficient to describe the full set of errors that a speech-to-text (STT) pipeline designed for transcription can make, such as those involving capitalisation, punctuation, and inverse text normalisation (ITN). It is therefore the case that improved WER does not always lead to increased productivity, and the inclusion of ASR in HIL may adversely affect productivity if it contains too many errors. Rev.com provides a convenient laboratory to explore these questions. Originally, the company provided transcriptions of audio and video content executed solely by humans (known as Revvers). More recently, ASR was introduced in an HIL workflow where Revvers postedited an ASR first draft. We provide an analysis of the interaction between metrics of ASR accuracy and the productivity of our 72,000+ Revvers transcribing more than 15,000 hours of media every week. To do this, we utilise two measures of transcriptionist productivity: transcriber real time factor (RTF) and words per minute (WPM). Through our work, we hope to focus attention on the human productivity and quality of experience (QoE) aspects of improvements in ASR and related technologies. Given the broad scope of content transcription applications and the still elusive objective of perfect machine performance, keeping the human in the loop in both practice and mind is critical. This paper provides an overview of human and machine transcription and Rev’s marketplace, followed by an analysis of the relationship between ASR accuracy and transcriptionist productivity, and concludes with suggestions for future work.

AI ≥ Journalism: How the Chinese Copyright Law Protects Tech Giants’ AI Innovations and Disrupts the Journalistic Institution

Journalism and other institutions clash over automated news generation, algorithmic distribution and content ownership worldwide. AI policies are the main mechanisms that establish and organise the hierarchies among these institutions. Few studies, however, have explored the normative dimension of AI in policymaking in journalism, especially beyond the West. This case study inspects the copyright law’s impact on AI innovation in newsrooms in the unexamined Chinese context. Using neo-institutional theory and policy network theory, the study investigates the Third Amendment to the Chinese Copyright Law, exemplary court cases regarding automated journalism copyright disputes (such as Tencent v. Yingxun and Film v. Baidu), and other supporting documents. The findings show how China’s copyright legal framework separates authorship and ownership; defines “originality” and “creativity” in human-machine collaboration; and prioritises tech companies while undermining journalistic autonomy. We argue that the law’s eager embrace of AI may give tech companies an advantage over news organisations that do not necessarily have a strategy to adopt AI. Moreover, it favours state-owned, resource-rich official media over the private sector. An implication of this shifting power dynamic is the possibility of privately owned news media being marginalised, resulting in even stronger state control over media production and information flow.

The risk of algorithm transparency: How algorithm complexity drives the effects on the use of advice

Although algorithmic decision support is omnipresent in many managerial tasks, a lack of algorithm transparency is often stated as a barrier to successful human–machine collaboration. In this paper, we analyze the effects of algorithm transparency on the use of advice from algorithms with different degrees of complexity. We conduct a set of laboratory experiments in which participants receive identical advice from algorithms with different levels of transparency and complexity. Our results indicate that not the algorithm itself, but the individually perceived appropriateness of algorithmic complexity moderates the effects of transparency on the use of advice. We summarize this effect as a plateau curve: While perceiving an algorithm as too simple severely harms the use of its advice, the perception of an algorithm as being too complex has no significant effect. Our insights suggest that managers do not have to be concerned about revealing algorithms that are perceived to be appropriately complex or too complex to decision‐makers, even if the decision‐makers do not fully comprehend them. However, providing transparency on algorithms that are perceived to be simpler than appropriate could disappoint people's expectations and thereby reduce the use of their advice.

Research on Hybrid Teaching Model Based on Human-Machine Collaboration

The information technology in education promotes the innovation of blended teaching mode with technical support. This paper constructs an online and offline hybrid teaching model based on the concept of human-computer collaboration. In addition, the experimental research method was used to investigate the effect of the hybrid teaching mode on the teaching effectiveness of the course "University Computer Fundamentals" as an example. The study found that learners who participated in the blended learning classroom performed better than traditional multimedia instruction, and that the different learning behaviors of learners correlated with their final grades. This shows that the online and offline hybrid teaching based on human-computer collaboration can improve the learning effectiveness of learners.

Sustainable Human–Machine Collaborations in Digital Transformation Technologies Adoption: A Comparative Case Study of Japan and Germany

The Digital Transformation (DX) megatrend is fundamentally disrupting and changing the nature of work, business, and industry at a rapid pace. Although the notion of DX has garnered much research interest from practitioners, scholarship on this topic is somehow lagging behind, possibly because of the lack of theoretical frameworks on DX. Recently, most Japanese firms have begun to use diverse digital technologies to sustain their competitive advantages. However, the return of investment on digital technologies has not been as high as expected for some firms. Furthermore, as the visions of Industry 5.0 describe sustainable, resilient, and human-centered future factories that will require smart and resilient capabilities both from next-generation manufacturing systems and human operators, it is necessary to design resilient human–machine collaborations within factories. To this end, this paper presents a research model between DX technologies and scientific problem-solving in terms of deduction, induction, and abduction inference structures as an approach to resilient human–machine collaborations. The purpose of this research is to analyze the difference in the utilization pattern of the digital technology of American, German, and Japanese firms based on three types of decision-making methods. Next, we apply this framework in a comparative case study of two Japanese firms and one German firm, where we find that there is a difference in DX technologies utilization among the Japanese and German firms. We assert that the utilization of IoT technology in the United States and Germany is pursuing IoT with the aim of autonomous control, whereas Japanese firms prioritize robot–human collaboration. Finally, we discuss how our findings contribute to the burgeoning field of resilient human–machine collaborations by showing the distinct roles of deduction, induction, and abduction inference structures. Furthermore, our research contributes to international comparative studies to identify the difference in national IT utilization. Lessons and implications are discussed.

Enabling human–machine collaboration in infrastructure inspections through mixed reality

In this study, we propose a novel end-to-end system called Human–Machine Collaborative Inspection (HMCI) to enable collaboration between inspectors with Mixed Reality (MR) headsets and a robotic data collection platform (robot) for structural inspections. We utilize the MR headset’s holographic display and precise head tracking to allow inspectors to visualize and localize information (e.g., structural defect) on the real scenes, which are gathered by the robot and processed by an offsite computational server. The primary use case of HMCI is to enable the inspector to visualize, supervise, and improve results produced by automated defect detection algorithms in near real-time. The workflow in HMCI starts with collecting images and depth data to generate 3D maps of the site from the robot. A technique called single-shot localization is developed to create visual anchors for real-time spatial alignment between the robot and the MR headset. The 3D map and images are then sent to the computational server for analysis to detect defects and their locations. Then, the information is received by the MR headset and overlaid on the actual scenes to visualize it with spatial context. An experimental study is conducted in a lab environment to demonstrate HMCI using Microsoft HoloLens 2 (HL2) as the MR headset and Turtlebot2 as the robot. We start with the reconstruction of a 3D environment using a 3D depth sensor (Azure Kinect) on Turtlebot2 and visually detect fiducial markers as regions-of-interest (replicating structural damage) along a predefined inspection path. Then, regions-of-interest are successfully anchored to the real scene and visualized through HL2. To our knowledge, HMCI is one of the first human–machine collaborative systems that can integrate robots and inspectors with the MR headset, which has been developed, tested, and presented for structural inspection.

Symbiosis Between D2D Communication and Industrial IoT for Industry 5.0 in 5G mm-Wave Cellular Network: An Interference Management Approach

Industrial Internet of Things (IIoT) paves way into Industry 5.0, which incorporates human–machine collaboration, thereby making manufacturing industry efficient. As 5G architecture supports massive IoT connectivity and has higher spectrum efficiency, device-to-device (D2D) communication is favorable at 28 GHz. While transmitting data from sensors to end user through IoT network, interference affects the system. Thus, an efficient resource allocation scheme is needed for minimizing interference and increasing data rate. Here, formulated problem is divided into two subproblems, channel assignment and power optimization in order to lower computational complexity. A partial resource multiplexing scheme is proposed that will allocate channels to available D2D users. Later, power optimization problem is formulated which is determined through Lagrangian dual optimization technique. Dynamic sectorization overcomes issue of increase in user traffic. Stability factor, fairness index (FI), and energy efficiency depict the performance superiority of proposed scheme over existing schemes. Simulation results prove efficacy of proposed system.

Human-machine collaboration using artificial intelligence to enhance the safety of donning and doffing personal protective equipment (PPE).

To compare the accuracy of monitoring personal protective equipment (PPE) donning and doffing process between an artificial intelligent (AI) machine collaborated with remote human buddy support system and an onsite buddy, and to determine the degree of AI autonomy at the current development stage. We conducted a pilot simulation study with 30 procedural scenarios (15 donning and 15 doffing, performed by one individual) incorporating random errors in 55 steps. In total, 195 steps were assessed. The human-AI machine system and the onsite buddy assessed the procedures independently. The human-AI machine system performed the assessment via a tablet device, which was positioned to allow full-body visualization of the donning and doffing person. The overall accuracy of PPE monitoring using the human-AI machine system was 100% and the overall accuracy of the onsite buddy was 99%. There was a very good agreement between the 2 methods (κ coefficient, 0.97). The current version of the AI technology was able to perform autonomously, without the remote human buddy's rectification in 173 (89%) of 195 steps. It identified 67.3% of all the errors independently. This study provides preliminary evidence suggesting that a human-AI machine system may be able to serve as a substitute or enhancement to an onsite buddy performing the PPE monitoring task. It provides practical assistance using a combination of a computer mirror, visual prompts, and verbal commands. However, further studies are required to examine its clinical efficacy with a diverse range of individuals performing the donning and doffing procedures.

Mensch-Maschine-Zusammenarbeit bei der Entscheidungsfindung

The use of ML-based decision support systems in business-related decision-making processes is a proven approach for companies to increase process performance and quality. To a certain extent, machines are capable of reproducing the cognitive abilities of humans in specific domains. In order to leverage the resulting potential, synergistic human-machine collaboration (HMC) is becoming increasingly important for companies. However, orchestrating HMC is dependent on a set of framework conditions that determine the success of the collaboration. This study examines the research question of how to utilize the concept of collaborative intelligence (CI) to enhance decision-making processes while using machine learning (ML) -based data prediction. The purpose is to identify success factors in the development, design, and implementation of an ML-based predictive analytics solution to orchestrate HMC in decision-making processes. These success factors state recommendations for companies to fulfil the necessary framework conditions for synergetic HMC orchestration. In total, five success factors were identified that represent a combination of theoretical findings and empirical insights. At the same time, further research needs were uncovered, which point out starting points for future research projects in the field of HMC. Â