Confidence Judgments Research Articles

Illusion of knowledge in statistics among clinicians: evaluating the alignment between objective accuracy and subjective confidence, an online survey

Healthcare professionals’ statistical illiteracy can impair medical decision quality and compromise patient safety. Previous studies have documented clinicians’ insufficient proficiency in statistics and a tendency in overconfidence. However, an underexplored aspect is clinicians’ awareness of their lack of statistical knowledge that precludes any corrective intervention attempt. Here, we investigated physicians’, residents’ and medical students’ alignment between subjective confidence judgments and objective accuracy in basic medical statistics. We also examined how gender, profile of experience and practice of research activity affect this alignment, and the influence of problem framing (conditional probabilities, CP vs. natural frequencies, NF). Eight hundred ninety-eight clinicians completed an online survey assessing skill and confidence on three topics: vaccine efficacy, p value and diagnostic test results interpretation. Results evidenced an overall consistent poor proficiency in statistics often combined with high confidence, even in incorrect answers. We also demonstrate that despite overconfidence bias, clinicians show a degree of metacognitive sensitivity, as their confidence judgments discriminate between their correct and incorrect answers. Finally, we confirm the positive impact of the more intuitive NF framing on accuracy. Together, our results pave the way for the development of teaching recommendations and pedagogical interventions such as promoting metacognition on basic knowledge and statistical reasoning as well as the use of NF to tackle statistical illiteracy in the medical context.

7- and 8-Year-Olds’ Struggle With Monitoring

Abstract: An often-replicated finding in metacognition research is that children overestimate their performance. To date, only a few studies have investigated the possible effects of item-specific feedback on metacognitive monitoring in young children. This study examined whether first-graders benefit from feedback to improve metacognitive monitoring discrimination when completing a paired-associates task for consistency. Over six sessions, N = 112 children evaluated whether they solved tasks correctly or not and gave item-specific confidence judgments. One group obtained only performance feedback; the other group received additional feedback on whether their performance matched their monitoring judgments. Results revealed that children could adequately discriminate between correct and incorrect answers in their confidence judgments. However, neither type of feedback improved metacognitive monitoring discrimination. We discuss the results against the theoretical background of Efklides’ self-regulation model and the cue utilization approach.

How Do Investor and Auditor Materiality Judgments Compare?

SUMMARY Materiality remains a challenging concept for auditors to implement in practice. The challenges underlying auditor materiality assessments are compounded by the fact that materiality is typically defined from the investor’s (rather than the auditor’s) perspective. Despite this investor orientation, there is little empirical evidence about investors’ materiality judgments, specific quantitative and qualitative factors underlying their judgments, and how their judgments compare to auditors who implement materiality in practice. This article summarizes a recent study by DeZoort, Holt, and Stanley (2019) that addresses this problem by modeling sophisticated and unsophisticated investors’ materiality judgments in a policy-capturing study, using experienced auditors as a benchmark. Results indicate significant differences in materiality judgments, judgment consensus, and cue utilization among the three participant groups. Findings also reveal between-group differences in self-reported quantitative materiality thresholds, judgment self-insight, as well as judgment confidence. We conclude the article with a discussion of the practical implications surrounding these findings.

A metacognitive approach to the study of motion-induced duration biases reveals inter-individual differences in forming confidence judgments.

Our ability to estimate the duration of subsecond visual events is prone to distortions, which depend on both sensory and decisional factors. To disambiguate between these two influences, we can look at the alignment between discrimination estimates of duration at the point of subjective equality and confidence estimates when the confidence about decisions is minimal, because observers should be maximally uncertain when two stimuli are perceptually the same. Here, we used this approach to investigate the relationship between the speed of a visual stimulus and its perceived duration. Participants were required to compare two intervals, report which had the longer duration, and then rate their confidence in that judgment. One of the intervals contained a stimulus drifting at a constant speed, whereas the stimulus embedded in the other interval could be stationary, linearly accelerating or decelerating, or drifting at the same speed. Discrimination estimates revealed duration compression for the stationary stimuli and, to a lesser degree, for the accelerating and decelerating stimuli. Confidence showed a similar pattern, but, overall, the confidence estimates were shifted more toward higher durations, pointing to a small contribution of decisional processes. A simple observer model, which assumes that both judgments are based on the same sensory information, captured well inter-individual differences in the criterion used to form a confidence judgment.

Philosophy instruction changes views on moral controversies by decreasing reliance on intuition

What changes people's judgments on moral issues, such as the ethics of abortion or eating meat? On some views, moral judgments result from deliberation, such that reasons and reasoning should be primary drivers of moral change. On other views, moral judgments reflect intuition, with reasons offered as post-hoc rationalizations. We test predictions of these accounts by investigating whether exposure to a moral philosophy course (vs. control courses) changes moral judgments, and if so, via what mechanism(s). In line with deliberative accounts of morality, we find that exposure to moral philosophy changes moral views. In line with intuitionist accounts, we find that the mechanism of change is reduced reliance on intuition, not increased reliance on deliberation; in fact, deliberation is related to increased confidence in judgments, not change. These findings suggest a new way to reconcile deliberative and intuitionist accounts: Exposure to reasons and evidence can change moral views, but primarily by discounting intuitions.

Hyperspectral Video Tracker Based on Spectral Deviation Reduction and a Double Siamese Network

The advent of hyperspectral cameras has popularized the study of hyperspectral video trackers. Although hyperspectral images can better distinguish the targets compared to their RGB counterparts, the occlusion and rotation of the target affect the effectiveness of the target. For instance, occlusion obscures the target, reducing the tracking accuracy and even causing tracking failure. In this regard, this paper proposes a novel hyperspectral video tracker where the double Siamese network (D-Siam) forms the basis of the framework. Moreover, AlexNet serves as the backbone of D-Siam. The current study also adopts a novel spectral–deviation-based dimensionality reduction approach on the learned features to match the input requirements of the AlexNet. It should be noted that the proposed dimensionality reduction method increases the distinction between the target and background. The two response maps, namely the initial response map and the adjacent response map, obtained using the D-Siam network, were fused using an adaptive weight estimation strategy. Finally, a confidence judgment module is proposed to regulate the update for the whole framework. A comparative analysis of the proposed approach with state-of-the-art trackers and an extensive ablation study were conducted on a publicly available benchmark hyperspectral dataset. The results show that the proposed tracker outperforms the existing state-of-the-art approaches against most of the challenges.

Using serial dependence to predict confidence across observers and cognitive domains.

Our perceptual system appears hardwired to exploit regularities of input features across space and time in seemingly stable environments. This can lead to serial dependence effects whereby recent perceptual representations bias current perception. Serial dependence has also been demonstrated for more abstract representations, such as perceptual confidence. Here, we ask whether temporal patterns in the generation of confidence judgments across trials generalize across observers and different cognitive domains. Data from the Confidence Database across perceptual, memory, and cognitive paradigms was reanalyzed. Machine learning classifiers were used to predict the confidence on the current trial based on the history of confidence judgments on the previous trials. Cross-observer and cross-domain decoding results showed that a model trained to predict confidence in the perceptual domain generalized across observers to predict confidence across the different cognitive domains. The recent history of confidence was the most critical factor. The history of accuracy or Type 1 reaction time alone, or in combination with confidence, did not improve the prediction of the current confidence. We also observed that confidence predictions generalized across correct and incorrect trials, indicating that serial dependence effects in confidence generation are uncoupled to metacognition (i.e., how we evaluate the precision of our own behavior). We discuss the ramifications of these findings for the ongoing debate on domain-generality versus domain-specificity of metacognition.

Gamified Inoculation Against Misinformation in India: A Randomized Control Trial

Although the spread of misinformation is a pervasive and disruptive global problem, extant research is skewed towards “WEIRD” countries leaving questions about how to tackle misinformation in the developing world with different media and consumption patterns unanswered. We report the results of a game-based intervention against misinformation in India. The game is based on the mechanism of psychological inoculation; borrowed from the medical context, inoculation interventions aim to pre-emptively neutralize falsehoods and help audiences spot and resist misinformation strategies. Though the efficacy of these games has been repeatedly demonstrated in samples from Western countries, the present study conducted in north India (n = 757) did not replicate earlier findings. We found no significant impact of the intervention on the perceived reliability of messages containing misinformation, confidence judgments, and willingness to share information with others. Our experience presents a teachable moment for the unique challenges associated with complex cultural adaptations and field work in rural areas. These results have significant ramifications for designing misinformation interventions in developing countries where misinformation is largely spread via encrypted messaging applications such as WhatsApp. Our findings contribute to the small but growing body of work looking at how to adapt misinformation interventions to cross-cultural settings.

Metacognition and Causal Inference in Audiovisual Speech

In multisensory environments, our brains perform causal inference to estimate which sources produce specific sensory signals. Decades of research have revealed the dynamics which underlie this process of causal inference for multisensory (audiovisual) signals, including how temporal, spatial, and semantic relationships between stimuli influence the brain's decision about whether to integrate or segregate. However, presently, very little is known about the relationship between metacognition and multisensory integration, and the characteristics of perceptual confidence for audiovisual signals. In this investigation, we ask two questions about the relationship between metacognition and multisensory causal inference: are observers' confidence ratings for judgments about Congruent, McGurk, and Rarely Integrated speech similar, or different? And do confidence judgments distinguish between these three scenarios when the perceived syllable is identical? To answer these questions, 92 online participants completed experiments where on each trial, participants reported which syllable they perceived, and rated confidence in their judgment. Results from Experiment 1 showed that confidence ratings were quite similar across Congruent speech, McGurk speech, and Rarely Integrated speech. In Experiment 2, when the perceived syllable for congruent and McGurk videos was matched, confidence scores were higher for congruent stimuli compared to McGurk stimuli. In Experiment 3, when the perceived syllable was matched between McGurk and Rarely Integrated stimuli, confidence judgments were similar between the two conditions. Together, these results provide evidence of the capacities and limitations of metacognition's ability to distinguish between different sources of multisensory information.

Preserved electrophysiological markers of confidence in schizophrenia spectrum disorder

A large number of behavioral studies suggest that confidence judgments are impaired in schizophrenia, motivating the search for neural correlates of an underlying metacognitive impairment. Electrophysiological studies suggest that a specific evoked response potential reflecting performance monitoring, namely the error-related negativity (ERN), is blunted in schizophrenia compared to healthy controls. However, attention has recently been drawn to a potential confound in the study of metacognition, namely that lower task-performance in schizophrenia compared to healthy controls involves a decreased index of metacognitive performance (where metacognitive performance is construed as the ability to calibrate one’s confidence relative to response correctness), independently of metacognitive abilities among patients. Here, we assessed how this confound might also apply to ERN-blunting in schizophrenia. We used an adaptive staircase procedure to titrate task-performance on a motion discrimination task in which participants (N = 14 patients and 19 controls) had to report their confidence after each trial while we recorded high density EEG. Interestingly, not only metaperceptual abilities were preserved among patients at the behavioral level, but contrary to our hypothesis, we also found no electrophysiological evidence for altered EEG markers of performance monitoring. These results bring additional evidence suggesting an unaltered ability to monitor perceptual performance on a trial by trial basis in schizophrenia.

The influence of metacognition monitoring on L2 Chinese audiovisual reading comprehension.

Metacognition monitoring is the ability to evaluate the cognitive process actively. L2 learners with high metacognition monitoring ability can better monitor reading processes and outcomes consciously, thus facilitating self-regulated learning and improving reading efficiency. Previous studies mostly used offline self-reports to examine the metacognition monitoring in static text reading by L2 learners. This study investigated the effects of different indicators of metacognition monitoring on L2 Chinese audiovisual comprehension by online confidence judgment and audiovisual comprehension tasks. Target measures of metacognition monitoring included absolute calibration accuracy based on video or test and relative calibration accuracy measured by Gamma or Spearman correlation coefficient. 38 intermediate-advanced Chinese learners participated in the study. Multiple regression analysis showed three main results. First, absolute calibration accuracy can significantly predict L2 Chinese audiovisual comprehension, while relative calibration accuracy has no significant effect. Second, the predictive effect of video-based absolute calibration accuracy is affected by the video difficulty, that is, the greater the video difficulty, the greater the impact on the performance of audiovisual comprehension. Third, the predictive effect of test-based absolute calibration accuracy is influenced by the language proficiency, specifically, the higher the L2 Chinese proficiency, the stronger the prediction on the performance of audiovisual comprehension. These results support a multidimensional view of metacognition monitoring by specifying how different indicators of metacognition monitoring may predict L2 Chinese audiovisual comprehension. The findings have important pedagogical implications for strategy training of metacognition monitoring and point to the necessity to take task difficulty and individual differences among learners into full consideration.

Korean dental hygiene students' perceptions and attitudes toward artificial intelligence: An online survey.

This study investigated Korean dental hygiene students' perceptions and attitudes toward artificial intelligence (AI) and aimed to identify needs for education to strengthen professional competencies. A 24-question online survey was conducted to the dental hygiene students from four Korean schools in 2021. The questionnaire included seven questions on basic characteristics and 17 AI-related questions on the student's attitudes toward AI, the confidence in AI, predictions about AI, and its future prospects. Responses were analyzed according to the frequencies and correlations between the participants' subjective level of knowledge about AI and questions using chi-square test. Invitations were sent out to 1310 students and 800 (61.1%) participated. Note that 44.2% of participants were interested in AI, and 93.1% accessed AI-related information through the internet. Participants expressed lower confidence in AI's diagnosis (14.8%) and judgment (8.1%) than in those of humans, and 21.9% believed AI would replace their job. The proportions of participants with positive perceptions of the usefulness and the potential for improvement of AI in dentistry were 65.5% and 55.4%, respectively. Participants from schools who had existing AI knowledge expressed higher demands for AI-related content as compared to those who did not (p < 0.05). Although dental hygiene students expressed low level of confidence in AI, they were interested in AI and had positive views of its application and potential for improvement. However, the fact they had little AI-related information from dental hygiene curriculum strongly suggests the need for AI-related lectures in schools to prepare for the future.

Neural activity tracking identity and confidence in social information.

Humans learn about the environment either directly by interacting with it or indirectly by seeking information about it from social sources such as conspecifics. The degree of confidence in the information obtained through either route should determine the impact that it has on adapting and changing behaviour. We examined whether and how behavioural and neural computations differ during non-social learning as opposed to learning from social sources. Trial-wise confidence judgements about non-social and social information sources offered a window into this learning process. Despite matching exactly the statistical features of social and non-social conditions, confidence judgements were more accurate and less changeable when they were made about social as opposed to non-social information sources. In addition to subjective reports of confidence, differences were also apparent in the Bayesian estimates of participants' subjective beliefs. Univariate activity in dorsomedial prefrontal cortex and posterior temporoparietal junction more closely tracked confidence about social as opposed to non-social information sources. In addition, the multivariate patterns of activity in the same areas encoded identities of social information sources compared to non-social information sources.

General knowledge norms: Updated and expanded for German.

The general knowledge questions introduced by Nelson and Narens (Journal of Verbal Learning and Verbal Behavior, 19(3), 338-368, 1980) have been a valuable research tool in various areas of cognitive research. We translated, updated, and expanded the set of questions for German. We present a total set of 356 general knowledge questions with their recall probability as well as metacognitive measures-confidence and peer judgments-based on a university student sample (N = 512). Furthermore, we present response latencies, pairwise correlations between recall probability and metacognitive judgments as well as the most common commission errors. These general knowledge questions can be used in studies with German speaking participants in a broad range of research fields, such as memory, illusory truth, misinformation, and metacognitive processes.

Comparing event-related potentials of retrospective and prospective metacognitive judgments during episodic and semantic memory

It is unclear whether metacognitive judgments are made on the basis of domain-generality or domain-specificity. In the current study, we compared both behavioral and event-related potential (ERP) correlates of retrospective (retrospective confidence judgments: RCJs), and prospective (feeling of knowing: FOK) metacognitive judgments during episodic and semantic memory tasks in 82 participants. Behavioral results indicated that FOK judgments reflect a domain-specific process, while RCJ reflect a domain-general process. RCJ and FOK judgments produced similar ERP waveforms within the memory tasks, but with different temporal dynamics; thus supporting the hypothesis that retrospective and prospective metacognitive judgments are distinct processes. Our ERP results also suggest that metacognitive judgments are linked to distributed neural substrates, rather than purely frontal lobe functioning. Furthermore, the role of intra-subject and inter-subject differences in metacognitive judgments across and within the memory tasks are highlighted.

A 3D mixed reality visualization of network topology and activity results in better dyadic cyber team communication and cyber situational awareness.

Cyber defense decision-making during cyber threat situations is based on human-to-human communication aiming to establish a shared cyber situational awareness. Previous studies suggested that communication inefficiencies were among the biggest problems facing security operation center teams. There is a need for tools that allow for more efficient communication of cyber threat information between individuals both in education and during cyber threat situations. In the present study, we compared how the visual representation of network topology and traffic in 3D mixed reality vs. 2D affected team performance in a sample of cyber cadets (N = 22) cooperating in dyads. Performance outcomes included network topology recognition, cyber situational awareness, confidence in judgements, experienced communication demands, observed verbal communication, and forced choice decision-making. The study utilized network data from the NATO CCDCOE 2022 Locked Shields cyber defense exercise. We found that participants using the 3D mixed reality visualization had better cyber situational awareness than participants in the 2D group. The 3D mixed reality group was generally more confident in their judgments except when performing worse than the 2D group on the topology recognition task (which favored the 2D condition). Participants in the 3D mixed reality group experienced less communication demands, and performed more verbal communication aimed at establishing a shared mental model and less communications discussing task resolution. Better communication was associated with better cyber situational awareness. There were no differences in decision-making between the groups. This could be due to cohort effects such as formal training or the modest sample size. This is the first study comparing the effect of 3D mixed reality and 2D visualizations of network topology on dyadic cyber team communication and cyber situational awareness. Using 3D mixed reality visualizations resulted in better cyber situational awareness and team communication. The experiment should be repeated in a larger and more diverse sample to determine its potential effect on decision-making.

Dynamics of task-based confidence in schizophrenia using seasonal decomposition approach

ObjectiveIntrospective Accuracy (IA) is a metacognitive construct that refers to alignment of self-generated accuracy judgments, confidence, and objective information regarding performance. IA not only refers to accuracy and confidence during tasks, but also predicts functional outcomes. The consistency and magnitude of IA deficits suggest a sustained disconnect between self-assessments and actual performance. The cognitive origins of IA are unclear and are not simply due to poor performance. We tried to capture task and diagnosis-related differences through examining confidence as a timeseries. MethodThis relatively large sample (N = 171; Bipolar = 71, Schizophrenia = 100) study used item by item confidence judgments for tasks including the Wisconsin Card Sorting Task (WCST) and the Emotion Recognition task (ER-40). Using a seasonal decomposition approach and AutoRegressive, Integrative and Moving Averages (ARIMA) time-series analyses we tested for the presence of randomness and perseveration. ResultsFor the WCST, comparisons across participants with schizophrenia and bipolar disorder found similar trends and residuals, thus excluding perseverative or random responding. However, seasonal components were weaker in participants with schizophrenia, reflecting a reduced impact of feedback on confidence. In contrast, for the ER40, which does not require identification of a sustained construct, seasonal, trend, and residual analyses were highly comparable. ConclusionSeasonal analysis revealed that confidence judgments in participants with schizophrenia on tasks requiring responses to feedback reflected diminished incorporation of external information, not random or preservative responding. These analyses highlight how time series analyses can specify potential faulty processes for future intervention.

Relationships between neuropsychological performance, insight, medication adherence, and social metacognition in schizophrenia

BackgroundSocial metacognition is still poorly understood in schizophrenia, particularly its neuropsychological basis and its impact on insight and medication adherence. We therefore quantified social metacognition as the agreement between objective and subjective mentalization and assessed its correlates in a sample of individuals with schizophrenia spectrum disorders. MethodsParticipants consisted of 143 patients with schizophrenia or schizoaffective disorders who underwent a metacognitive version of a mentalization task, an extensive neuropsychological battery, and a clinical evaluation to assess their insight into illness and medication adherence. We studied potential interactions between confidence judgments and several neuropsychological and clinical variables on mentalization accuracy with mixed-effects multiple logistic regressions. ResultsConfidence judgments were closely associated with mentalization accuracy, indicative of good social metacognition in this task. Working memory, visual memory, and reasoning and problem-solving were the three neuropsychological dimensions positively associated with metacognition. By contrast, the two measures of medication adherence were associated with poorer metacognition, whereas no association was found between metacognition and clinical insight. The multiple regression model showed a significant positive impact of better working memory, older age at onset, longer duration of hospitalization, and worse medication adherence on social metacognition. ConclusionsWe discuss possible mechanisms underlying the apparent association between social metacognition and working memory. Adherence should be monitored when remediating social metacognition, and psychoeducation should be given to patients with a high level of awareness of their capacity to mentalize.

“I guess it was more than just my general knowledge of chemistry”: exploring students’ confidence judgments in two-tiered assessments

Two-tiered assessment structures with paired content and confidence items are frequently used within chemistry assessments to stimulate and measure students’ metacognition. The confidence judgment is designed to promote students’ reflection on their application of content knowledge and can be characterized as calibrated or miscalibrated based on their accuracy. Previous studies often attributed students’ miscalibrated confidence rankings to metaignorance, however, in this qualitative study, interviews with general chemistry students were thematically analysed to provide a more robust understanding of the processes and factors students use when engaging with these metacognitive prompts in a chemistry assessment. Both calibrated and miscalibrated confidence judgments were observed independent of accuracy. Students who provided miscalibrated confidence judgments often used unreliable metrics such as processing fluency which can mimic content mastery whereas students who provided more accurate evaluations of their confidence relied more heavily on their stable understanding of chemistry concepts. Many students cited previous experiences, underlying self-efficacy beliefs, and/or the use of test-taking strategies which negatively or positively impacted their confidence. These findings suggest that the confidence tier is indeed capturing students’ self-assessment, however, students’ confidence judgments are based on a range of factors independent of content knowledge which may impede on the utility of this metacognitive tool for students, researchers, and instructors.

Exploring the Relation Between Metacognition, Gender, and Personality in Colombian University Students

In the present study, the relationship between objective and subjective measures of metacognition, personality traits of the Big Five (extraversion, agreeableness, conscientiousness, neuroticism, and openness) and gender were examined. A convenience sample of 352 university students completed the Metacognitive Awareness Inventory, the Adjectives to Evaluate Personality instrument, and completed 3 domain-specific tests (vocabulary, probabilities, paper folding) along with confidence in performance judgments for each item on these tests. Through a combination of descriptive statistics, zero-order bivariate correlations, simultaneous multiple regression, and a multivariate analysis of covariance, findings indicate that objective and subjective measures of metacognition are weakly related and that subjective measures of metacognition were more strongly related to personality traits. Conscientiousness and openness were the only personality traits that positively predicted metacognition. Gender affected both subjective and objective measures of metacognition, even after controlling for university type (private, public) and perceptions of academic performance (high, low) such that males were more accurate and less biased in their monitoring than females only in mathematical reasoning and they reported higher awareness of their knowledge and regulation of cognition than their female counterparts. Findings support the need to better understand how personality traits and gender affect self-regulated learning skills like metacognition to improve educational practices.