Accurate Mental Models Research Articles

Evaluating What Others Say: The Effect of Accuracy Assessment in Shaping Mental Models of AI Systems

Forming accurate mental models that align with the actual behavior of an AI system is critical for successful user experience and interactions. One way to develop mental models is through information shared by other users. However, this social information can be inaccurate and there is a lack of research examining whether inaccurate social information influences the development of accurate mental models. To address this gap, our study investigates the impact of social information accuracy on mental models, as well as whether prompting users to validate the social information can mitigate the impact. We conducted a between-subject experiment with 39 crowdworkers where each participant interacted with our AI system that automates a workflow given a natural language sentence. We compared participants' mental models between those exposed to social information of how the AI system worked, both correct and incorrect, versus those who formed mental models through their own usage of the system. Specifically, we designed three experimental conditions: 1) validation condition that presented the social information followed by an opportunity to validate its accuracy through testing example utterances, 2) social information condition that presented the social information only, without the validation opportunity, and 3) control condition that allowed users to interact with the system without any social information. Our results revealed that the inclusion of the validation process had a positive impact on the development of accurate mental models, especially around the knowledge distribution aspect of mental models. Furthermore, participants were more willing to share comments with others when they had the chance to validate the social information. The impact of inaccurate social information on altering user mental models was found to be non-significant, while 69.23% of participants incorrectly judged the social information accuracy at least once. We discuss the implications of these findings for designing tools that support the validation of social information and thereby improve human-AI interactions.

Exploring Users' Mental Models and Privacy Concerns During Interconnected Interactions

Users frequently use their smartphones in combination with other smart devices, for example, when streaming music to smart speakers or controlling smart appliances. During these interconnected interactions, user data gets handled and processed by several entities that employ different data protection practices or are subject to different regulations. Users need to understand these processes to inform themselves in the right places and make informed privacy decisions. We conducted an online survey (N=120) to investigate whether users have accurate mental models about interconnected interactions. We found that users consider scenarios more privacy-concerning when multiple devices are involved. Yet, we also found that most users do not fully comprehend the privacy-relevant processes in interconnected interactions. Our results show that current privacy information methods are insufficient and that users must be better educated to make informed privacy decisions. Finally, we advocate for restricting data processing to the app layer and better encryption to reduce users' data protection responsibilities.

Do textbooks provide opportunities to develop meaningful botanical literacy? A case study of the scientific model of plant nutrition

ABSTRACT Botanical literacy is a crucial component of science education, particularly in light of the contemporary socio-ecological challenges. However, the literature reports that, due to the phenomenon known as Plant Awareness Disparity (PAD), average botanical literacy is scant. Multiple factors influence the effectiveness of botanical education, including the quality of instructional materials teachers employ to support their instruction. Therefore, this study assesses the treatment of the Scientific Model of Plant Nutrition (SMPN) in seven commonly used Spanish Biology textbooks for secondary education to determine whether they foster a meaningful botanical literacy and empower students with the scientific competence needed to understand and make informed decisions regarding plant-related global challenges. The investigation evaluates four fundamental aspects: structure, conceptual content, images, and activities. Findings reveal that the analysed textbooks present PAD features; promote reproductive and superficial learning that fails to foster the construction of interrelated and scientifically accurate mental models of plant nutrition; include visual representations that do not always enhance learning; and propose tasks which are insufficient to develop the scientific skills and competence pursued by current science teaching-learning. The implications of this research on botanical literacy and teaching-learning of the SMPN are discussed, and measures to enhance educational materials are proposed.

Recycling water: board membership and industry-peer pressures

PurposeDespite the well-recognized importance of recycled water, the study of industry-peer pressure on recycled water is relatively new. This study investigates how organizations experience and react to industry-peer pressure to set recycled water targets. Additionally, this study investigates the role of board chairs involved in sustainability committees in contributing to responses to industry-peer pressure.Design/methodology/approachUsing Eviews 12, this study employed a pooled logistic regression model to analyze data from 1,346 firms on Taiwan and Taipei exchanges (2017–2020).FindingsThe findings revealed that frequency-based imitation drives recycled water target-setting diffusion. However, there is no direct relationship between outcome-based imitation and recycled water target-setting. Notably, outcome-based imitation drives the adoption of recycled water target-setting of firms with board-chair membership in sustainability committees.Research limitations/implicationsThis study faces certain data limitations. First, this study primarily focuses on water recycling. Future research could explore other ways to reduce water usage, such as using water-efficient equipment. Second, this study gathered information solely on the presence or absence of a board chairperson on the sustainability committee. Future researchers could explore the impact of the composition of sustainability committee on recycled water target-setting. Lastly, the sample used in this study is restricted to Taiwan's corporations that existed during 2017–2020. Future researchers may consider adopting a longitudinal design in other economies to address this limitation.Practical implicationsThe findings of this study offer several guidelines and implications for recycled water target-setting and the composition of sustainability committees. It responds to an urgent call for solutions to water shortages when pressure from governments and nongovernmental organizations is relatively absent. The number of industry peers that have already set recycled water targets is indispensable for motivating firms to set their own recycled water targets. In terms of insufficient water-related regulatory pressure and normative pressure, this study found evidence suggesting that the direct motivation for setting recycled water targets stems from mimetic pressures via frequency-based imitation. The evidence in this study suggests that policymakers should require companies to disclose their peers’ recycled water target information, as doing so serves as an alternative means to achieving SDG 6.3.Social implicationsRecycled water target-setting might be challenging. Water recycling practices may face strong resistance and require substantial additional resources (Zhang and Tang, 2019; Gao et al., 2019; Gu et al., 2023). Therefore, this study suggests that firms should ensure the mindfulness of board members in promoting the welfare of the natural environment when making recycled water target-setting decisions. To reap the second-mover advantage, firms must consider the conditions in which board members can more effectively play their role. Corporations may help their chairpersons in setting recycled water targets by recruiting them as members of sustainability committees. Meanwhile, chairpersons tend to activate accurate mental models when the water conservation performance of pioneering industry peers is strong enough to indicate the potential benefits of adopting recycled water target-setting. Investors’ and stakeholders’ understanding of how the composition of sustainability committees is related to recycled water target-setting may help to identify the potential drivers of firms’ water responsibility. Investors and stakeholders should distinguish firms in terms of the board chair’s membership of their sustainability committee and focus on water-use reduction outcomes in the industry. This study provides insights into circumstances whereby chairpersons help to restore the water ecosystem.Originality/valueThis study explains how frequency-based and outcome-based imitation are two prominent mechanisms underlying the industry-peer pressure concerning recycled water target-setting. Moreover, this study fills literature gaps related to the moderating roles of board-chair membership in sustainability committees concerning industry-peer pressure on recycled water target-setting.

Assessing Training Methods for Advanced Driver Assistance Systems and Autonomous Vehicle Functions: Impact on User Mental Models and Performance

Understanding the complexities of Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicle (AV) technologies is critical for road safety, especially concerning their adoption by drivers. Effective training is a crucial element in ensuring the safe and competent operation of these technologies. This study emphasises the critical role of training methodologies in shaping drivers’ mental models, defined as an individual’s cognitive frameworks for understanding and interacting with ADAS and AV systems. Their mental models substantially influence their interactions with those technologies. A comparative analysis of text-based and video-based training methods has been conducted to assess their influence on participants’ performance and the development of their mental models of ADAS and AV functionalities. Performance is evaluated in terms of the accuracy and reaction time of the participants as they interacted with ADAS and AV functions in a driving simulation. The findings reveal that video-based training yielded better performance outcomes, more accurate mental models, and a deeper understanding of ADAS functionalities among participants. These findings are crucial for policy makers, automotive manufacturers, and educational institutions involved in driver training. They underscore the necessity of developing tailored training programs to facilitate the proficient and safe operation of increasingly complex automotive technologies.

Block Cookies, Not Websites: Analysing Mental Models and Usability of the Privacy-Preserving Browser Extension CookieBlock

In the modern web, users are confronted with a plethora of complex privacy-related decisions about cookies and consent, often com- pounded by misleading policies and deceptive patterns. Past efforts to enhance online privacy have failed due to their dependence on website compliance. A solution to this lies in privacy-enhancing tools that are directly controlled by the user. However, challenges related to the usability and flawed understanding of the tools’ func- tionality hinder their widespread adoption. To address this problem, we evaluated the browser extension CookieBlock as an example of a current tool, which supports users by blocking tracking cookies independent of website compliance. We used a complementary approach consisting of an expert eval- uation of CookieBlock and the related tools NoScript and Ghostery, and a laboratory user study focusing on the unique details of how users interact with CookieBlock specifically. The laboratory study with 42 participants investigated usage, mental models, and us- ability of CookieBlock based on eye tracking, interaction, and self- report data. While CookieBlock received good usability ratings, 18 participants were unable to solve a website breakage caused by cookie misclassification on their own. Overall, the results revealed flawed mental models of CookieBlock’s functionality and resulting challenges in making the connection between website breakage and cookie misclassification. Implications for CookieBlock and related applications include interface design recommendations supporting accurate mental models and the proposal of improved heuristics to better guide users and warn them about potential identified website breakage.

Computational Thinking and Notional Machines: The Missing Link

In learning to program and understanding how a programming language controls a computer, learners develop both insights and misconceptions whilst their mental models are gradually refined. It is important that the learner is able to distinguish the different elements and roles of a computer (compiler, interpreter, memory, etc.), which novice programmers may find difficult to comprehend. Forming accurate mental models is one of the potential sources of difficulty inextricably linked to mastering computing concepts and processes, and for learning computer programming. It is common to use some form of representation (e.g., an abstract machine or a Computational Agent (CA)) to support technical or pedagogic explanations. The Notional Machine (NM) is a pedagogical device that entails one or more computational concepts, originally described as an idealised computer operating with the constructs of a particular programming language. It can be used to support specific or general learning goals and will typically have some concrete representation that can be referred to. Computational Thinking (CT), which is defined as a way of thinking that is used for [computational] problem solving, is often presented as using a CA to carry out information processing presented by a solution. In CT, where the typical goal is to produce an algorithm or a computer program, the CA seemingly serves a purpose very similar to an NM. Although it changes through the different stages of development (of the learner and of the curriculum), the roles of CAs and NMs can be seen as versatile tools that connect a learner’s mental model with the conceptual model of a program. In this article, we look at this relationship between CAs and NMs, and indicate how they would look at different stages of learning. We traverse the range of definitions and usages of these concepts, and articulate models that clarify how these are viewed in the literature. This includes exploring the nature of machines and agents, and how historical views of these relate to modern pedagogy for computation. We argue that the CA can be seen as an abstract, simplified variant of an NM that provides a useful perspective to the learner to support them to form robust mental models of NMs more efficiently and effectively. We propose that teaching programming should make use of the idea of a CA at different stages of learning, as a link that connects a learner’s mental model to a full NM.

Modeling Adaptive Expression of Robot Learning Engagement and Exploring Its Effects on Human Teachers

Robot Learning from Demonstration (RLfD) allows non-expert users to teach a robot new skills or tasks directly through demonstrations. Although modeled after human–human learning and teaching, existing RLfD methods make robots act as passive observers without the feedback of their learning statuses in the demonstration gathering stage. To facilitate a more transparent teaching process, we propose two mechanisms of Learning Engagement , Z2O-Mode and D2O-Mode, to dynamically adapt robots’ attentional and behavioral engagement expressions to their actual learning status. Through an online user experiment with 48 participants, we find that, compared with two baselines, the two kinds of Learning Engagement can lead to users’ more accurate mental models of the robot’s learning progress, more positive perceptions of the robot, and better teaching experience. Finally, we provide implications for leveraging engagement expression to facilitate transparent human-AI (robot) communication based on our key findings.

Co-Designing with Users the Explanations for a Proactive Auto-Response Messaging Agent

Explanations of AI Agents' actions are considered to be an important factor in improving users' trust in the decisions made by autonomous AI systems. However, as these autonomous systems evolve from reactive, i.e., acting on user input, to proactive, i.e., acting without requiring user intervention, there is a need to explore how the explanation for the actions of these agents should evolve. In this work, we explore the design of explanations through participatory design methods for a proactive auto-response messaging agent that can reduce perceived obligations and social pressure to respond quickly to incoming messages by providing unavailability-related context. We recruited 14 participants who worked in pairs during collaborative design sessions where they reasoned about the agent's design and actions. We qualitatively analyzed the data collected through these sessions and found that participants' reasoning about agent actions led them to speculate heavily on its design. These speculations significantly influenced participants' desire for explanations and the controls they sought to inform the agents' behavior. Our findings indicate a need to transform users' speculations into accurate mental models of agent design. Further, since the agent acts as a mediator in human-human communication, it is also necessary to account for social norms in its explanation design. Finally, user expertise in understanding their habits and behaviors allows the agent to learn from the user their preferences when justifying its actions.

Mental models of driver monitoring systems: Perceptions of monitoring capabilities in an online U.S-based sample

Driver monitoring, whether accompanying vehicle automation or not, appears poised for wide deployment in the near future. Driver monitoring is itself a form of automation (often needed in response to having automated other aspects of driving), and previous automation research has indicated that accurate mental models are critical to successful human-automation interactions. In this study, a descriptive survey of car owners examined perceived capabilities of generic driver monitoring systems (DMS) to detect 30 different potentially distracting behaviors or adverse mental states. Additional open-ended questions queried drivers about beliefs regarding how DMS work and how they might be defeated. Exploratory factor analysis revealed that perceived capabilities factored into four categories, including beliefs that driver monitoring systems can detect: (1) console and control interactions; (2) smartphone interactions; (3) gaze position and in-cabin movement; and (4) adverse mental states. Although research and industry analysis have suggested that camera-based gaze monitoring will be the path forward, a considerable minority of drivers appear to believe that console interactions will be monitored. Further, car owners in this study generally were skeptical about the ability DMS to detect adverse mental states, which may be in misalignment with the capabilities of systems that will be available in the near future. For future research, questions remain about the potential implications of false alarms (real and perceived) in driver monitoring, as well as the role of experience and initial training/instruction for deployed systems.

Longer-term exposure vs training: Their effect on drivers' mental models of ADAS technology

Drivers often learn about the advanced driver assistance systems (ADAS) on their vehicles over time and through trial and error. While this experience can aid drivers’ understanding about the systems, it may not necessarily lead to sufficient and accurate mental models, especially concerning less frequent “edge case” situations. This study recruited 39 new owners of vehicles equipped with ADAS technology to which the owners were naïve. The initial mental model of these owners was evaluated using a mental model assessment. To understand changes in mental models over time the assessment was repeated six times over the course of approximately 6 months. Weekly mileage, technology usage, and information regarding their exposure to edge case scenarios was also collected. At the end of the 6 months, participants completed a simulator drive using adaptive cruise control (ACC) that included several edge cases. Over the course of the first 6 months of vehicle ownership, drivers’ scores on the mental model assessment improved. These improvements were largely due to increased understanding of the technology’s limitations as opposed to improvements in knowledge about system function. With respect to driving performance in the simulator session, the mental model scores were not predictive of responses to the edge cases. However, a comparison of those mental model scores against weak and strong mental model benchmark scores gathered in a previous study revealed that mental models improve over 6 months (for some drivers), but not to the level of understanding of a group that received a short but extensive introduction to ACC. This suggests that there is room for improvement in how drivers gain understanding about driver support features and further underscores the need of training and education for proper use and interactions.

Usability testing of three visual HMIs for assisted driving: How design impacts driver distraction and mental models

There is a variety of visual human-machine interfaces (HMI) designed across vehicle manufacturers that support drivers while supervising driving automation features, such as adaptive cruise control (ACC). These various designs communicate the same limited amount of information to drivers about their ACC system and it is unclear which HMI designs impact driver distraction the least or how their design could be modified to help drivers develop more accurate mental models of their ACC system. Using a user-centred design (UCD) approach, we designed a speedometer to inform drivers about some of the system’s capabilities and then invited 23 drivers to use ACC in a low-fidelity driving simulator to compare the usability of three HMIs using eye-tracking, response times, and qualitative data. Our attempt at designing an intuitive and more informative speedometer received mixed results, but design recommendations are given regarding the indication of the set target speed, set time gap between vehicles (headway distance), and system mode (conventional or adaptive cruise). Practitioner summary: Manufacturers’ heterogeneous designs of their visual HMIs for the ACC systems may impact driver distraction in different ways. We used usability testing to compare three HMIs in a driving simulator and make several design recommendations to indicate speed, time gap, and system mode in a more efficient way. Abbreviations: ACC: adaptive cruise control; ADAS: advanced driving assistance system; HMI: human-machine interface; ISO: international organisation for standardization; OEM: original equipment manufacturer; RSME: rating scale of mental effort; RT: response time; R-TLX: raw task load index; SUS: system usability scale; TGT: total glance time; UCD: user-centred design; UX: user experience; xTGT: extended total glance time

“I’m Afraid I Can’t Do That, Dave”; Getting to Know Your Buddies in a Human–Agent Team

The rapid progress in artificial intelligence enables technology to more and more become a partner of humans in a team, rather than being a tool. Even more than in human teams, partners of human–agent teams have different strengths and weaknesses, and they must acknowledge and utilize their respective capabilities. Coordinated team collaboration can be accomplished by smartly designing the interactions within human–agent teams. Such designs are called Team Design Patterns (TDPs). We investigated the effects of a specific TDP on proactive task reassignment. This TDP supports team members to dynamically allocate tasks by utilizing their knowledge about the task demands and about the capabilities of team members. In a pilot study, agent–agent teams were used to study the effectiveness of proactive task reassignment. Results showed that this TDP improves a team’s performance, provided that partners have accurate knowledge representations of each member’s skill level. The main study of this paper addresses the effects of task reassignments in a human–agent team. It was hypothesized that when agents provide explanations when issuing and responding to task reassignment requests, this will enhance the quality of the human’s mental model. Results confirmed that participants developed more accurate mental models when agent-partners provide explanations. This did not result in a higher performance of the human–agent team, however. The study contributes to our understanding of designing effective collaboration in human–agent teams.

HMM-based models of control room operator's cognition during process abnormalities. 1. Formalism and model identification

Operators' mental models play a central role in safety-critical domains like the chemical process industries. Accurate mental models, i.e., a correct understanding of the process and its causal linkages, are prerequisites for safe operation. Mental models are often defined and explained in abstract terms that make their interpretation subjective and prone to bias. In this work, we propose a Hidden Markov Model (HMM) based formalism to characterize control room operators' mental models while handling abnormal situations. We show that a suitable HMM representing the operator's mental model – including the states, state transition probabilities, and emission probability distributions – can be identified experimentally using data of the operator's control actions, eye gaze, and process variable values. This HMM can be used for the quantitative assessment of operators' mental models as illustrated using various case studies. We discuss the potential applications of the model in identifying various cognitive errors and human reliability assessment. In Part 2 of this paper, we use the proposed approach to assess operators' learning during training.

Advanced vehicle technology: Mapping mental model accuracy and system exposure to driver behavior

Advanced driver assistance systems have potential to increase safety and comfort for drivers; however, drivers need to understand the capabilities and limitations of these systems to use them appropriately. This study sought to explore how the quality (accuracy) of drivers’ mental models of adaptive cruise control (ACC) impacted their behavior and interactions while using the system. Seventy-eight participants drove in a high-fidelity driving simulator while operating an ACC system, in normal conditions and while interacting with the system interface. Participants with stronger (more accurate) mental models glanced to the road ahead more often during normal conditions early on compared to drivers were less accurate mental models; however, these differences diminished with increased system exposure. Glance behavior while interacting with the system and time to complete the interactions were less effected by the strength of the participant’s mental model. Results are discussed in the context of driver education and training.

The Effects of Team Mental Model Complexity on Team Information Search and Performance Trajectories

Drawing on the concept of requisite complexity, we propose that mental model complexity is crucial for teams to thrive in dynamic complex environments. Using a longitudinal research design, we examined the influence of team mental model complexity on team information search and performance trajectories in a sample of 64 teams competing in a business strategy simulation over time. We found that team information search positively influences performance growth over time. More specifically, and consistent with requisite complexity, we found that mental model complexity positively influences both performance growth and information search over time, above and beyond the effects of mental model similarity and accuracy.

How far in the future can we predict others’ affective states?

IntroductionHuman social interactions are rooted in the ability to understand and predict one’s own and others emotions. Individuals develop accurate mental models of emotional transitions (MMET) by observing regularities in affective experiences (DOI: 10.1073/pnas.1616056114) and a failure in this regard can produce maladaptive behaviors, one of the hallmark features in several psychiatric conditions.ObjectivesTo investigate whether MMET are stable over time and which emotion dimensions (e.g., valence, dominance) influence MMET over time.MethodsWe selected thirty-seven emotion categories (DOI: 10.1177/0539018405058216) and five different time intervals (from 15 minutes to 4 days). Sixty-two healthy participants rated the likelihood of transition between all possible pairs of affective states at each time interval.ResultsAs expected, we observed a trend toward uncertainty as the timescale increased. In addition, the probability of shifting between two affective states having the same valence (e.g., happiness and contentment) was rated higher than for emotions with opposite polarity (e.g., happiness and sadness). Even though this pattern becomes gradually noisier for predictions far in the future, it is still present for infradian intervals (Fig.1).ConclusionsOur results suggest that MMET are informed by the valence dimension and moderately influenced by the timescale of the prediction. These findings in the healthy population may prompt the exploration of emotion dynamics in psychiatric conditions. Future studies could leverage the MMET approach to test whether specific psychiatric disorders (e.g., bipolar disorder) are associated with abnormal patterns of emotion transitions.DisclosureNo significant relationships.

The Effect of a Concept Mapping Intervention on Shared Cognition and Adaptive Team Performance Over Time

Research has demonstrated the value of team adaptation for organizational teams. However, empirical work on interventions that teams can take to increase adaptive team performance is scarce. In response, this study proposes a concept mapping intervention as a way to increase teams’ ability to adapt following a task change. Particularly, this study examines the effect of a concept mapping intervention on team transition adaptation (the drop in performance after a change) and reacquisition adaptation (the slope of performance after the change) via its effect on task mental models and transactive memory systems. We conducted a longitudinal experimental study of 44 three-person teams working on an emergency management simulation. Findings suggest that the concept mapping intervention promotes reacquisition adaptation, task mental models, and transactive memory systems. Results also suggest that task mental models mediate the effect of the concept mapping intervention on reacquisition adaptation. A post hoc analysis suggests that the concept mapping intervention is only effective if it leads to high task mental model accuracy. Our study presents concept mapping as a practical intervention to promote shared cognition and reacquisition adaptation.

People accurately predict the transition probabilities between actions.

Social life is a complex dance. To coordinate gracefully with one's partners, one must predict their actions. Here, we investigated how people predict others' actions. We hypothesized that people can accurately predict others' future actions based on knowledge of their current actions, coupled with knowledge of action transitions. To test whether people have accurate knowledge of the transition probabilities between actions, we compared actual rates of action transitions-calculated from four large naturalistic datasets-to participants' ratings of the transition probabilities between corresponding sets of actions. In five preregistered studies, participants demonstrated accurate mental models of action transitions. Furthermore, we found that people drew upon conceptual knowledge of actions-described by the six-dimensional ACT-FASTaxonomy-to guide their accurate predictions. Together, these results indicate that people can accurately anticipate other people's moves in the dance of social life and that the structure of action knowledge may be tailored to making these predictions.

Towards a manager's mental model: Conceptual change through business simulation

Mental models are relevant in both managerial decision making and management education. Decision-making studies have demonstrated how managers benefit from more accurate mental models by gaining higher-quality heuristics and better performance. However, management education studies assessing mental models as learning outcomes remain rare. This study aims at understanding how students' mental models change during a business simulation course. We designed a pre- and post-simulation concept map assignment that illustrated students' mental models as number of concepts, connections between concepts, and written explanations. The findings of our study demonstrate that students’ mental models changed through two interconnected processes: addition of new knowledge and change of misconceptions in existing knowledge. Based on these results, we composed a model of conceptual change through business simulation gaming. Our study contributes to the research on learning from business simulations and addresses the recent call for more conceptual change research in social sciences by enhancing the conceptual change discussion in the management education literature.