Response Confidence Ratings Research Articles

Influence of Sex and Age on Ratings of Confidence and Relevance in Continuing Certification Longitudinal Assessment: A Pilot Study.

Longitudinal assessments use spaced repetition of items to facilitate learning. Algorithms selecting repetition items can prioritize various properties for future presentation. The purpose of this pilot study was to evaluate the relationship between participant ratings of item-specific confidence and/or practice relevance and participant age, sex, and response correctness. This is a prospective quality improvement study of 403 American Board of Physical Medicine and Rehabilitation diplomates with time-limited certificates. Participants answered 20 items quarterly over four quarters, rating each item on its relevance to their practice and their confidence in their response. The relationship between sex and ratings of response confidence was significant, with women less likely than men to be confident in their responses, regardless of correctness. Younger physicians were significantly more confident in their responses and rated items as more practice relevant. Women physicians were less confident than men in their item-specific confidence ratings, regardless of correctness, on the American Board of Physical Medicine and Rehabilitation continuing certification longitudinal knowledge assessment. Older physicians were less confident in their responses than younger physicians. The findings supported the American Board of Physical Medicine and Rehabilitation prioritization of response correctness and practice relevance, rather than response confidence, to select items for spaced repetition in American Board of Physical Medicine and Rehabilitation's continuing certification longitudinal assessment.

Examining response confidence in multiple text tasks

Students’ confidence in their responses to a multiple text-processing task and their justifications for those confidence ratings were investigated. Specifically, 215 undergraduates responded to two academic questions, differing by type (i.e., discrete and open-ended) and by domain (i.e., developmental psychology and astrophysics), using a digital library of seven varied texts (e.g., journal article, newspaper, Wikipedia). Students then indicated how confident they were in their responses and provided justifications for their level of confidence. This investigation had four main goals: (a) to determine the strength of students’ response confidence; (b) to understand the criteria students use to derive their confidence ratings; (c) to consider the effect of task conditions on ratings and justifications; and (d) to examine the relation between students’ response confidence ratings and justifications and the accuracy and quality of their responses. Students’ justifications for response confidence were found to vary across task conditions. Further justifications for response confidence based on sources consulted (i.e., text-directed justifications) were found to relate to response accuracy for the discrete question and to response quality for the open-ended question.

Effects of retinal disparity and screen format on three-dimensional object position and course-prediction judgments

A human factors study in which participants viewed 3-D imagery under two task conditions designed to mimic the critical information processing elements of an aircraft control application is reported. In one task, observers judged the relative distance between objects in a 3-D workspace, whereas the other task required observers to extrapolate object positions from course heading cues. Operator performance was assessed for imagery presented with or without retinal disparity cues in three display formats (i.e. , plan view, simple perspective view, and enhanced perspective view) and at three signal-to-clutter levels. The findings indicate that retinal disparity cues reduce the number and magnitude of response errors in the course prediction task, but do not affect observer's response times or ratings of response confidence. The effects of display format varied between the two tasks. In the relative distance assessment task, response times for the simple perspective format were shorter than those for the plan view or enhanced perspective formats. In the course prediction task, response times for the plan view and simple perspective formats were shorter than for the enhanced perspective format. Display format did not affect response errors, error magnitude, or ratings of response confidence.

Responding to Feedback after Multiple-choice Answers: The Influence of Response Confidence

A model is proposed that describes how people choose multiple-choice answers and deal with subsequent feedback. Central to the model is the construct of likelihood, a signal that: (1) indexes the match between task demands and what is known, (2) gives a basis for decisions, and (3) is estimated by response confidence ratings. In two experiments, subjects answered an item, rated confidence, and studied feedback (for 100 items), and then answered the items again. All predictions of the model were confirmed. Specifically, for right/wrong feedback messages, feedback study intervals were longer after correct than incorrect choices (for correct choices, these intervals decreased as confidence increased). Also, there was no relation between the probability of correcting errors and level of confidence. For feedback messages that consisted of an item with correct alternative marked, there was a positive relation between probability of correcting errors and level of confidence.

Studies in infant behavioural audiometry. II. Six-week-old infants.

The behaviour of 90 clinically normal 6-week-old babies previously studied as neonates was video-recorded while they were presented with sequences of sound stimuli. The stimuli varied in sound pressure level, bandwidth and rise time, and included two voice signals and a no-sound (control) trial. Video records were made both for the whole body aspect and for a 3 1/2 X 'close-up' of the head. Later, the video records were shown to 6 observers who were allowed to see the babies for 10 s at each trial. The first 5 s was a prestimulus observation period, and the second 5 s usually contained a sound stimulus. Between the trials, the observers were given 20 s in which to record (i) prestimulus activity, (ii) confidence in response, and (iii) movement details (data for [iii] not reported here). Different segments of the baby's body were observed on different occasions as detailed in our neonatal report. The response confidence ratings were analysed, using aspects of signal detection theory, to show differences amongst various body segments (p less than 0.001), sound pressure levels (SPL) (p less than 0.001), band widths (BW) (p less than 0.001), and rise times (RT) (p less than 0.001). There were significant interactions between SPL and BW (p less than 0.05) and SPL X RT (p less than 0.001). The 3 factor interaction (SPL X BW X RT) was also significant (p less than 0.001). A 90-dB broad-spectrum noise band was by far the most effective stimulus. The response to different sound stimuli was affected by the baby's posture (lying or sitting) and by prestimulus activity state. The results are discussed in relation to our previous neonatal study.

Studies in infant behavioural audiometry. I. Neonates.

The behaviour of 93 clinically normal neonates was video-recorded while they were presented with sequences of sound stimuli which varied in sound pressure level, bandwidth and rise time and which included two voice signals and a no-sound (control) trial. Video records were made both for the whole body aspect and for a 3 1/2 X "close-up" of the head. Later, the video records were shown to 6 observers who were allowed to see the babies for 10 sec at each trial. The first 5 sec was a pre-stimulus observation period, and the second 5 sec usually contained a sound stimulus. Between the trials, the observers were given 20 sec in which to record (1) pre-stimulus activity, (2) confidence in response, and (3) movement details (data for (3) not reported here). For the whole-body aspect, observers were allowed to see (1) the whole body, (2) the head, arms and trunk, (3) the arms, trunk and legs, (4) the head only, (5) the arms and trunk only, or (6) the legs only, during different viewing sessions. For the head aspect, they could see (7) the whole head, (8) the head above the upper lip, or (9) the head below the upper lip. This segmentation was achieved by masking parts of the television screen with shutters. The response confidence ratings were analysed using aspects of signal detection theory to show differences amongst various body segments (p less than 0.001), sound pressure levels (p less than 0.001), bandwidths (p less than 0.001), and rise times (p less than 0.01). There were significant interactions between sound pressure level X bandwidth (p less than 0.001), sound pressure level X rise time (p less than 0.01), and bandwidth X rise time (p less than 0.001). A 90-dB broad-spectrum noiseband was by far the most effective stimulus. The response to different sound stimuli was differentially affected by pre-stimulus activity state. The results are discussed in relation to recent work on clinical screening techniques.