Metacognitive Monitoring Ability Research Articles

The effect of calibration training on the calibration of intelligence analysts' judgments

AbstractExperts are expected to make well‐calibrated judgments within their field, yet a voluminous literature demonstrates miscalibration in human judgment. Calibration training aimed at improving subsequent calibration performance offers a potential solution. We tested the effect of commercial calibration training on a group of 70 intelligence analysts by comparing the miscalibration and bias of their judgments before and after a commercial training course meant to improve calibration across interval estimation and binary choice tasks. Training significantly improved calibration and bias overall, but this effect was contingent on the task. For interval estimation, analysts were overconfident before training and became better calibrated after training. For the binary choice task, however, analysts were initially underconfident and bias increased in this same direction post‐training. Improvement on the two tasks was also uncorrelated. Taken together, results indicate that the training shifted analyst bias toward less confidence rather than having improved metacognitive monitoring ability.

The relationship between metacognitive monitoring, non-verbal intellectual ability, and memory performance in kindergarten children

When assessing their certainty, children are often poor at accurately monitoring their level of learning. The study examined the relationships between memory performance, intellectual ability, and metacognitive monitoring accuracy in kindergarten children. We also explored whether specific thresholds in memory performance and non-verbal intellectual ability influence metacognitive monitoring accuracy to identify group-specific patterns that might be masked by an overall linear analysis. We assessed the monitoring discrimination of 290 kindergarteners (Mage 6 years) using a paired-associates learning task. Results showed small correlations between task performance, intellectual ability, and metacognitive monitoring. Non-verbal intellectual ability provided explanatory value for monitoring accuracy beyond memory performance. We observed group-specific results consistent with the unskilled-and-unaware effect; children with the highest memory skills were more effective at discriminating between correct and incorrect answers than their peers with the lowest memory skills. However, kindergarteners with the highest non-verbal intellectual abilities did not demonstrate greater cognitive adaptability in novel tasks, as their monitoring accuracy was comparable to that of peers with average or lower intellectual abilities. Findings indicate that both task performance and non-verbal intellectual ability are relevant for monitoring accuracy, but the impact of non-verbal intellectual ability was less significant than anticipated. The modest correlation suggests that kindergarteners' non-verbal intellectual ability and metacognitive monitoring abilities operate relatively independently.

Can you accurately monitor your behaviors while multitasking? The effect of multitasking on metacognition.

With the pace of life accelerating, multitasking has become the norm in daily life. According to research, multiple cognitive processes, including numerical reasoning, comprehension, and writing, are negatively affected by multitasking. However, only a few studies have investigated the relationship between multitasking and metacognition. In this study, the effect of multitasking on metacognition was examined using a prospective monitoring paradigm (prediction of subsequent recall performance). In Experiment 1, the participants simultaneously studied word pairs (primary task) and differentiated between different sound pitches (secondary task) and then predicted their performance in a subsequent memory test for the studied word pairs (prospective metacognitive monitoring). The accuracy of metacognitive evaluation with multitasking was then compared with that without multitasking. In Experiment 2, sounds and icons of real-life applications were used to improve the ecological validity of the experiment in the secondary task. The results indicated that multitasking impaired metacognition in both artificial and real-life simulated scenarios. In addition, the participants who engaged in more media multitasking in their daily lives exhibited poorer metacognitive monitoring abilities in single tasks.

Negative emotion can be "more negative" for those with high metacognitive abilities when problem-solving.

Metacognitive monitoring ability enables you to learn and solve problems more efficiently through appropriate strategies. At the same time, those who are high in monitoring ability are known to allocate more cognitive resources to the perception and control of negative emotions, as compared to those with low metacognitive ability. Therefore, while monitoring emotions may help reduce the negative emotion by enabling efficient control, it could also interrupt the use of an efficient strategy when problem-solving, as cognitive resources may be depleted. To confirm this, we divided participants into groups with high and low monitoring abilities and manipulated emotions by presenting emotional videos. Subsequent to the manipulation, problem solving strategies were examined using items from the Cognitive Reflection Test (CRT). Results showed that those who were high in monitoring ability were shown to use more efficient problem-solving strategies than those who were lower in monitoring ability, but only in situations when positive or no emotions were manipulated. However, as hypothesized, when negative emotion was aroused, the CRT scores of high monitoring ability group were significantly lowered, decreasing to the same performance as those with low monitoring ability. We also found that metacognitive monitoring ability, when interacting with emotion, indirectly affected CRT scores, and that monitoring and control, when affected by emotion, were mediated in the process. These findings suggest a novel and complicated interaction between emotion and metacognition and warrant further research.

I know that I know. But do I know that I do not know?

Metacognition-the ability of individuals to monitor one's own cognitive performance and decisions-is often studied empirically based on the retrospective confidence ratings. In experimental research, participants are asked to report how sure they are in their response, or to report how well their performance in high-level cognitive or low-level perceptual tasks is. These retrospective confidence ratings are used as a measure of monitoring effectiveness: larger difference in confidence ratings assigned to correct and incorrect responses reflects better ability to estimate the likelihood of making an error by an experiment participant, or better metacognitive monitoring ability. We discuss this underlying assumption and provide some methodological consideration that might interfere with interpretation of results, depending on what is being asked to evaluate, how the confidence response is elicited, and the overall proportion of different trial types within one experimental session. We conclude that mixing trials on which decision confidence is assigned when positive evidence needs to be evaluated and the trials on which absence of positive evidence needs to be evaluated should be avoided. These considerations might be important when designing experimental work to explore metacognitive efficiency using retrospective confidence ratings.

Metacognitive Illusions: A Positivity Effect in Judgments of Learning for Older but Not Younger Adults

The positivity effect for metacognitive judgments (judgments of learning, JOLs) of emotional words in recognition memory was shown in older adults, in contrast to younger adults, who typically displayed the emotional salience effect. This is compatible with the socioemotional selection theory, which suggests the presence of a positive stimulus bias in older adults' cognitive processes. This study examined whether the positivity effect and age-related differences could be extended to a picture study to determine whether the positivity effect in older adults is robust in the metacognitive domain. Younger and older adults studied negative, positive, and neutral pictures, followed by JOLs and then a recognition test that asked participants to judge whether the picture was shown in the studying stage or not. Age-related differences were found not only in recognition memory performance for emotional pictures but also in JOLs and their accuracy. Younger adults showed an emotional salience effect for both memory performance and JOLs. Older adults' JOLs showed a positivity effect, but their actual memory performance was influenced by emotion, and this inconsistency between metacognitive judgments and memory performance is a metacognitive illusion. These findings support the cross-material replicability of a positivity bias in older adults in the metacognitive domain and suggest that we should be cautioned about the detrimental effects of this metacognitive illusion in older adults. It illustrates an age difference in the effect of emotion on individual metacognitive monitoring ability.

Exploring psychometric properties of children' metacognitive monitoring

Two independent data sets assessing children's metacognitive monitoring abilities were used to explore the psychometric properties of classical and often-used monitoring measures in primary school age. Theoretically, monitoring is an overarching skill that helps individuals evaluate task mastery, strategy use, and correctness of performance. Monitoring skills are increasingly targeted when addressing individual differences in scholastic achievement and intervention approaches to foster students' self-regulated learning early on. In such contexts, knowledge about central psychometric properties is essential. Results of both studies revealed high internal consistency of prospective and retrospective monitoring judgments. When equivalent item sets (in terms of item difficulty) were considered (Study 1), split-half reliabilities were also satisfying. However, analyses revealed that the monitoring judgments' reliability depends on the reliability of the first-order task (recognition memory test). Retesting children of Study 2 after six months revealed considerable fluctuations in the monitoring measures. Among the included monitoring measures, reliabilities of within-person correlations (Gammas) between performance and confidence and recognition response times and confidence were poorest. Results are discussed in the context of the underlying theoretical construct and implications for research and practice.

Exploring the interplay between attributions and metacognitive monitoring ability in a post-secondary classroom

Understanding the development of self-regulated learning (SRL) in applied educational contexts is currently an important goal for researchers. There exists a relatively rich literature for most SRL components in isolation yet the field is lacking in understanding their coordination. This study examined the relationship between metacognitive monitoring and causal attributions in a classroom setting over the duration of an academic semester. Undergraduates (N = 193) enrolled in an educational psychology course provided monitoring judgments before and after four tests and also rated their attributional responses after each test. A number of interesting findings emerged. Attributions were not found to be significant predictors of subsequent judgments of monitoring accuracy. Monitoring accuracy measured with an absolute accuracy index yielded minimal significant relationships with attributional dimensions yet a prediction bias index was found to be associated with locus of causality, stability, and personal control and postdiction bias with external control, stability, and personal control. Finally, internal attributions led to higher performance, supporting prior literature, however more stable attributions were also related to higher performance, a finding not supported by prior literature. Findings and implications are discussed with regard to the facilitation of effective academic self-regulation.

Evidence for Age-Equivalent and Task-Dissociative Metacognition in the Memory Domain.

Research suggests that metacognitive monitoring ability does not decline with age. For example, judgments-of-learning (JOL) accuracy is roughly equivalent between younger and older adults. But few studies have asked whether younger and older adults’ metacognitive ability varies across different types of memory processes (e.g., for items vs. pairs). The current study tested the relationship between memory and post-decision confidence ratings at the trial level on item (individual words) and associative (word pairs) memory recognition tests. As predicted, younger and older adults had similar metacognitive efficiency, when using meta-d’/d’, a measure derived from Signal Detection Theory, despite a significant age effect favoring younger adults on memory performance. This result is consistent with previous work showing age-equivalent metacognitive efficiency in the memory domain. We also found that metacognitive efficiency was higher for associative memory than for item memory across age groups, even though associative and item recognition memory (d’) were statistically equivalent. Higher accuracy on post-test decision confidence ratings for associative recognition relative to item recognition on resolution accuracy itself (meta-d’) and when corrected for performance differences (meta-d’/d’) are novel findings. Implications for associative metacognition are discussed.

Metacognitive Monitoring and Mathematical Abilities: Cognitive Diagnostic Model and Signal Detection Theory Approach

Besides various in-class assessments, there exist some standardized assessment tools that are administered in several countries, such as PISA (Programme for International Student Assessment) and TIMMS (Trends in International Mathematics and Science Study). The questions’ contents, type of responding, grading, and the analyses in these large-scale tests have been diversified in years. In this study, it was aimed to identify the abilities that are measured at PISA mathematics test in a single testing procedure and by utilizing the methods of analyses of Cognitive Diagnostic Model (CDM) as well as Signal Detection Theory (SDT), which have not been used so far in the assessment of these abilities. Therefore, a randomly selected sample of 6th-grade students ( N =230) in Izmir was tested with a PISA-equivalent 12-item mathematics test, where the items are graded dichotomously (correct vs. incorrect). CDM estimates were calculated by using the Deterministic Input Noisy Output and Gate (DINA) Model. The participants were asked to report whether they thought they could solve the question correctly, guess even if they thought they could not solve the question, and then, rate their confidence levels on the correctness of their answers in turn so as to allow us to measure their “metacognitive monitoring performance” with the SDT method, which refers to the ability to differentiate correct and incorrect responses. In short, a better metacognitive monitoring performance was obtained by measuring how well once could differentiate their correct and incorrect responses with the observation of they prefer reporting and then giving high confidence levels to the actually correct responses and prefer passing to give an answer yet rate lower confidence levels to the actually incorrect responses given as pure guesses. The results showed that CDM fits well to the assessment of PISA test and those who were better at the ability of “reasoning and developing strategies” in particular among four possible abilities detected with CDM (“representing and communicating”, “mathematization”, “reasoning and developing strategies”, “using symbolic and technical language”) had also better metacognitive monitoring performance. The present study, therefore, contributes to the research that investigates what features the ability of better differentiating correct and incorrect responses are actually linked. Based on the results, it is suggested that a better metacognitive monitoring ability is linked to having a better ability of “reasoning and developing strategies” in particular. Additionally, it is suggested that measuring metacognitive monitoring performance at PISA -or even any other possible tests- with the SDT calculation method, that has a relatively straightforward testing procedure, may yield various estimates for the students’ abilities measured at the test as well as their related higher-order abilities.

Predictors of Adults’ Metacognitive Monitoring Ability: The Roles of Task and Item Characteristics

Learners’ monitoring judgments during reading are based on a variety of cues, and the roles of task features in promoting and constraining judgment accuracy are beginning to be understood. This work examined task and item characteristics influencing adults’ monitoring of performance during reading and study tasks. In Study 1a, adults (N = 242) read an expository text, provided text ratings, completed comprehension items, and provided confidence judgments during testing. Readers’ text complexity ratings predicted their item-level confidence, while their ratings of interest and cohesion predicted their bias scores. In Study 1b, the roles of linguistic features in predicting item-level monitoring estimates were examined using the comprehension items (item n = 23) from Study 1a. Selected item features, including length, ease, density, concreteness, and meaningfulness, were examined. Item ease significantly predicted item-level bias, while ease, density, concreteness, and meaningfulness predicted item-level confidence. Study 2 (N = 68) applied similar procedures to examine the effect of item feedback on the contributions of item features to learners’ monitoring. Participants were randomly assigned to one of two item feedback conditions. Item density emerged as a significant predictor of item-level confidence, but only for those who first received item-by-item correctness feedback followed by massed feedback. Implications for future research are discussed.

Metacognitive Monitoring of Text Comprehension: An Investigation on Postdictive Judgments in Typically Developing Children and Children With Reading Comprehension Difficulties.

The ability to assess and monitor one’s own understanding of a written text is fundamental for learning and academic achievement. In the current paper, postdictive monitoring of text comprehension (i.e., the ability to judge the accuracy of responses previously given to a reading comprehension test) was investigated in both typically developing (TD) children and children with reading comprehension difficulties. Children from primary school (3rd to 5th grade) and secondary school (6th to 8th grade) participated in the study (N = 245). They were administered standardized tasks for reading comprehension, in which they had to read two texts and answer 12 multiple-choice questions after each text; subsequently, they had to provide postdictive judgments evaluating their performance: for each answer they had to select whether they judged it as correct, incorrect or whether they were uncertain. Two scores were calculated: Bias score, indicating the difference between metacognitive judgments of accuracy and actual performance; and Accurate estimation, indicating the sum of correct answers judged as “correct” and incorrect answers judged as “incorrect.” Results showed that primary school children were more overconfident than secondary school children and made fewer Accurate estimations especially for “correct” responses. Furthermore, the consideration of a group of children with reading comprehension difficulties showed that these failures are linked to worse metacognitive monitoring ability of comprehension performance in comparison not only to age-matched controls but also to the TD group of third-graders. Implications for learning and achievement are discussed.

Unskilled but subjectively aware: Metacognitive monitoring ability and respective awareness in low-performing students.

Two studies were conducted to further examine the unskilled-and-unaware effect and to test whether low-performing students are indeed unaware of their (expected) lower metacognitive monitoring abilities. Postdicted judgments of performance and second-order judgments (SOJs) were solicited to test students' metacognitive awareness. Given that global and local judgments tend to differ (the confidence-frequency effect), we investigated whether students' (un)awareness pertains to both types of judgments. A first study focusing on global judgments was conducted in a regular exam setting with 196 undergraduate education students. A second study with 115 undergraduate education students examined both global and local judgments. Local judgments were analyzed on an average level and according to different signal detection theory categories (hits, correct rejections, misses, and false alarms). In both studies, students were grouped in four performance quartiles. The results showed that low-performing students highly overestimated their performance (they were functionally overconfident). However, their SOJs indicated that they were less confident in their judgments than the other students, and thus seemed to be aware of their low ability to estimate their own performance (they were not subjectively overconfident). This was observed for global as well as for averaged local SOJs. Moreover, an analysis of the local judgments revealed that students' SOJs varied depending not only on whether their judgments were accurate but also on whether or not they thought they knew the answer to an item. In sum, SOJs provide valuable information about students' metacognitive awareness.

Knowing right from wrong in mental arithmetic judgments: calibration of confidence predicts the development of accuracy.

Does knowing when mental arithmetic judgments are right—and when they are wrong—lead to more accurate judgments over time? We hypothesize that the successful detection of errors (and avoidance of false alarms) may contribute to the development of mental arithmetic performance. Insight into error detection abilities can be gained by examining the “calibration” of mental arithmetic judgments—that is, the alignment between confidence in judgments and the accuracy of those judgments. Calibration may be viewed as a measure of metacognitive monitoring ability. We conducted a developmental longitudinal investigation of the relationship between the calibration of children's mental arithmetic judgments and their performance on a mental arithmetic task. Annually between Grades 5 and 8, children completed a problem verification task in which they rapidly judged the accuracy of arithmetic expressions (e.g., 25+50 = 75) and rated their confidence in each judgment. Results showed that calibration was strongly related to concurrent mental arithmetic performance, that calibration continued to develop even as mental arithmetic accuracy approached ceiling, that poor calibration distinguished children with mathematics learning disability from both low and typically achieving children, and that better calibration in Grade 5 predicted larger gains in mental arithmetic accuracy between Grades 5 and 8. We propose that good calibration supports the implementation of cognitive control, leading to long-term improvement in mental arithmetic accuracy. Because mental arithmetic “fluency” is critical for higher-level mathematics competence, calibration of confidence in mental arithmetic judgments may represent a novel and important developmental predictor of future mathematics performance.

Metacognitive judgments-of-learning in adolescents with autism spectrum disorder

This study investigated metacognitive monitoring abilities in adolescents with autism spectrum disorder in two experiments using the judgment-of-learning paradigm. Participants were asked to predict their future recall of unrelated word pairs during the learning phase. Experiment 1 compared judgments-of-learning made immediately after learning and judgments-of-learning made after a delay. We found that both groups overestimated their memory performance but that overall there were no group differences in judgment-of-learning accuracy. Additionally, both groups displayed the standard delayed judgment-of-learning effect (yielding greater judgment accuracy in delayed compared to immediate judgments), suggesting that both groups were able to use appropriate information in making their judgments-of-learning. Experiment 2 assessed whether adolescents with autism spectrum disorder could regulate their study time according to their judgments-of-learning using a self-paced learning procedure. Results showed that both groups spent more time learning items given lower judgments-of-learning. Finally, Experiment 2 showed that judgments-of-learning and study time varied according to item difficulty in both groups. As a whole, these findings demonstrate that adolescents with autism spectrum disorder can accurately gauge their memory performance while learning new word associations and use these skills to control their study time at learning.

Retrospective and prospective metacognitive judgments in rhesus macaques (Macaca mulatta)

A growing body of research suggests that some non-human animals are capable of making accurate metacognitive judgments. In previous studies, non-human animals have made either retrospective or prospective judgments (about how they did on a test or how they will do on a test, respectively). These two types of judgments are dissociable in humans. The current study tested the abilities of two rhesus macaque monkeys to make both retrospective and prospective judgments about their performance on the same memory task. Both monkeys had been trained previously to make retrospective confidence judgments. Both monkeys successfully demonstrated transfer of retrospective metacognitive judgments to the new memory task. Furthermore, both monkeys transferred their retrospective judgments to the prospective task (one, immediately, and one, following the elimination of a response bias). This study is the first to demonstrate both retrospective and prospective monitoring abilities in the same monkeys and on the same task, suggesting a greater level of flexibility in animals' metacognitive monitoring abilities than has been reported previously.

Metacognitive awareness and adaptive recognition biases.

Prior literature has primarily focused on the negative influences of misleading external sources on memory judgments. This study investigated whether participants can capitalize on generally reliable recommendations in order to improve their net performance; the focus was on potential roles for metacognitive monitoring (i.e., knowledge about one's own memory reliability) and performance feedback. In Experiment 1, participants received explicit external recommendations (Likely Old or Likely New) that were 75% valid during recognition tests containing deeply and shallowly encoded materials. In Experiment 2, participants received recommendations of differing validity (65% and 85%). Discrimination improved across both experiments when external recommendations were present versus absent. This improvement was influenced by metacognitive monitoring ability measured in the absence of recommendations. Thus, effective incorporation of external recommendations depended in part on how sensitive observers were to gradations of their internal evidence when recommendations were absent. Finally, corrective feedback did not improve participants' ability to use external recommendations, suggesting that metacognitive monitoring ability during recognition is not easily improved via feedback.

Metacognitive monitoring abilities and amyloid deposition imaging in normal elderly individuals

Metacognitive monitoring abilities and amyloid deposition imaging in normal elderly individuals