Final Test Performance Research Articles

Automatic noise detection for ambulatory electrocardiogram in presence of ventricular arrhythmias through a machine learning approach

Noise detection in ambulatory electrocardiography is investigated as a machine learning binary classification problem on a set of twelve noise indices. Ten of these noise indices are replicated from relevant scientific literature. Two novel noise indices are also introduced: the electrocardiogram derivative pattern similarity index (edp) and the number of inversions of the signal’s slope over a minimum amplitude threshold (inv). Noise indices were computed on ex-novo labeled data, which included ventricular arrhythmias such as ventricular tachycardia, flutter, and fibrillation. Signal quality labels were assigned on windows of 2 seconds, from which a set of labels on 10-second windows was also created. Six classification models were trained on 2 s and 10 s window data with a multi-stage optimization procedure. A feature selection stage was performed for each model, probing the usefulness of different noise indices, including noise and ventricular arrhythmias. Hyperparameter selection and model assessment were conducted through nested cross-validation. Final test performance metrics reached values as high as Matthew’s correlation coefficient of 0.854 and balanced accuracy of 0.923. Particular attention was placed on the correct quality classification of records with ventricular arrhythmias, and test specificity on records with ventricular arrhythmias was as high as 0.976. This study provided new insights into the role of different noise indices in a challenging context with ventricular arrhythmias.

Prior knowledge and new learning: An experimental study of domain-specific knowledge.

It is commonly claimed that higher domain knowledge enhances new learning-the knowledge-is-power hypothesis. However, a recent meta-analysis (Simonsmeier et al., 2022) has challenged this idea, finding no overall relationship between prior knowledge and new learning across hundreds of highly variable effect sizes. The authors note that this variability and lack of randomized controlled experiments preclude broad claims regarding the influence of prior knowledge on learning. The present study (conducted in 2020) provides an experimental assessment of the causal effect of prior domain knowledge on new learning. Participants were randomly assigned to receive training in one of two academic domains over 3 days before learning new information about topics in both domains for a later test. Training was specific to three of four topics within that domain, allowing the untrained topic in the trained domain to act as a measure of new learning in that domain. New learning, measured as final test performance or knowledge gains, did not differ between the high and low domain knowledge conditions. Experimentally induced prior domain knowledge did not affect new learning. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

교수를 통한 학습(learning-by-teaching)이 메타인지적 판단에 미치는 영향

The present study investigated the effect of learning-by-teaching on learners' metacognitive judgments and examined how beneficial effects of teaching changed depending on whether or not retrieval practice was evoked during teaching. Undergraduate students studied science passages through two learning phases. In the first phase participants read the passage and in the second phase they either restudied them (control), created a video lecture with their notes (teaching without retrieval practice; TnRP), or taught without any materials (teaching with retrieval practice; TRP). Following each phase, they made metacognitive judgments regarding their comprehension of the passages, and then took a final test, which included retention and inference questions. The results showed that, despite the non-significant difference in the final test performance, there were significant differences in the judgments of learning (JOLs) across the three conditions. After the second learning phase, JOLs increased in the control group, remained unchanged in the TnRP group, and decreased in the TRP group, compared to their initial JOLs. As a result, the TRP group showed significantly lower levels of metacognitive bias and higher metacomprehension accuracy compared to the other groups. Our findings suggest that learning-by-teaching can mitigate overconfidence in learners and enhance metacomprehension accuracy, particularly when teaching is accompanied by retrieval practice.

Spaced Retrieval Practice: Can Restudying Trump Retrieval?

We investigated spaced retrieval and restudying in 3 preregistered, online experiments. In all experiments, participants studied 40 Swahili–English word pair translations during an initial study phase, restudied intact pairs or attempted to retrieve the English words to Swahili cues twice in three spaced practice sessions, and then completed a final cued-recall test. All 5 sessions were separated by 2 days. In Experiment 1, we manipulated the response format during retrieval (covert vs. overt) and the test list structure (blocked vs. intermixed covert/overt retrieval trials). A memory rating was required on all trials (retrieval: “Was your answer correct?”; restudy: “Would you have remembered the correct translation?”). Response format had no effect on recall, but surprisingly, final test performance for restudied items exceeded both the overt and covert retrieval conditions. In Experiment 2, we manipulated the requirement to make a memory rating. If a memory rating was required, final test restudy performance exceeded retrieval performance, replicating Experiment 1. However, the pattern was descriptively reversed if no rating was required. In Experiment 3, the memory rating was removed altogether, and we examined recall performance for items restudied versus retrieved once, twice, or thrice. Performance improved with practice, and retrieval performance exceeded restudy performance in all conditions. The reversal of the typical retrieval practice effect observed in Experiments 1 and 2 is discussed in terms of theories of reactivity of memory judgments.

Do Dictation Privileges Add Value to Radiology Electives If Medical Students Have Access to Online Modules?

Rationale and ObjectivesTraditional in-person medical student radiology electives are predominantly observerships. However, during the COVID-19 restrictions, many schools across Canada and the United States moved their radiology electives to an online format with great success. To the best of our knowledge, an evaluation of student experiences at a site where the 2-week on-site elective has both dictation privileges and an online component has not been completed. Materials and MethodsWe analyzed the pre- and post-test data from the online component of a radiology elective, retrieved the total number of dictation reports, and reviewed qualitative student feedback to gauge student engagement and learning. We used a generalized linear mixed model (GLMM) to compare the difference between the student scores on pre- and postmodule section quizzes, accounting for individual and section-specific variation in performance. A second GLMM was fit to assess improvement in the cumulative test taken before and after the entire elective. Both models accounted for the potential benefit of dictation. ResultsStudent score improved significantly on the postsection tests (βpost = 15.9% ± 0.86% SE, P < 0.001) and on the final postelective test (βpost = 28.1% ± 1.8% SE, P < 0.001). The effect of total number of dictated studies on final test performance was not significant (βpost = 0.07% ± 0.07% SE, P = 0.317). ConclusionThis retrospective study suggests that resources should be directed to the development of mixed online/in-person electives for ideal student engagement at the senior medical student level. Further work must be completed to understand the potential benefits of, and barriers to, student dictation at the medical student level.

Battling Unawareness of One's Test Performance: Do Practice, Self-Efficacy, and Emotional Intelligence Matter?

The "Dunning-Kruger effect" refers to the tendency of poor performers to overestimate test outcomes. Although a widespread phenomenon, questions exist regarding its source and sensitivity to countermeasures. The present field study aimed to (a) examine whether practice with tests used in previous classes can enhance students' ability to estimate test outcomes, (b) determine the main source of the effect (i.e., is it unawareness of one's readiness or wishful thinking?), and (c) assess the extent to which particular individual differences can be used as predictors of test performance. In this study, participants practiced with old tests and then completed the final exam. Before and after the exam, they predicted their grades and indicated their subjective confidence in the predictions made. Furthermore, participants' emotional intelligence and self-efficacy about their academic abilities were surveyed. Results suggested that poor performers were not unaware of their test preparation, but rather engaged in wishful thinking. In fact, although they overestimated their test grades, their estimates not only improved after completing the final test but also were regarded with little confidence. Overall, estimation bias was a good predictor of students' final test performance, whereas subjective confidence and emotional intelligence only weakly predicted such performance. Thus, if proactive interventions are to be developed for at-risk students, performance-estimation tasks may offer valuable information regarding such students' future performance in a course much more than emotional intelligence and self-efficacy measures.

Development and Validation of a Model to Identify Critical Brain Injuries Using Natural Language Processing of Text Computed Tomography Reports

Clinical text reports from head computed tomography (CT) represent rich, incompletely utilized information regarding acute brain injuries and neurologic outcomes. CT reports are unstructured; thus, extracting information at scale requires automated natural language processing (NLP). However, designing new NLP algorithms for each individual injury category is an unwieldy proposition. An NLP tool that summarizes all injuries in head CT reports would facilitate exploration of large data sets for clinical significance of neuroradiological findings. To automatically extract acute brain pathological data and their features from head CT reports. This diagnostic study developed a 2-part named entity recognition (NER) NLP model to extract and summarize data on acute brain injuries from head CT reports. The model, termed BrainNERD, extracts and summarizes detailed brain injury information for research applications. Model development included building and comparing 2 NER models using a custom dictionary of terms, including lesion type, location, size, and age, then designing a rule-based decoder using NER outputs to evaluate for the presence or absence of injury subtypes. BrainNERD was evaluated against independent test data sets of manually classified reports, including 2 external validation sets. The model was trained on head CT reports from 1152 patients generated by neuroradiologists at the Yale Acute Brain Injury Biorepository. External validation was conducted using reports from 2 outside institutions. Analyses were conducted from May 2020 to December 2021. Performance of the BrainNERD model was evaluated using precision, recall, and F1 scores based on manually labeled independent test data sets. A total of 1152 patients (mean [SD] age, 67.6 [16.1] years; 586 [52%] men), were included in the training set. NER training using transformer architecture and bidirectional encoder representations from transformers was significantly faster than spaCy. For all metrics, the 10-fold cross-validation performance was 93% to 99%. The final test performance metrics for the NER test data set were 98.82% (95% CI, 98.37%-98.93%) for precision, 98.81% (95% CI, 98.46%-99.06%) for recall, and 98.81% (95% CI, 98.40%-98.94%) for the F score. The expert review comparison metrics were 99.06% (95% CI, 97.89%-99.13%) for precision, 98.10% (95% CI, 97.93%-98.77%) for recall, and 98.57% (95% CI, 97.78%-99.10%) for the F score. The decoder test set metrics were 96.06% (95% CI, 95.01%-97.16%) for precision, 96.42% (95% CI, 94.50%-97.87%) for recall, and 96.18% (95% CI, 95.151%-97.16%) for the F score. Performance in external institution report validation including 1053 head CR reports was greater than 96%. These findings suggest that the BrainNERD model accurately extracted acute brain injury terms and their properties from head CT text reports. This freely available new tool could advance clinical research by integrating information in easily gathered head CT reports to expand knowledge of acute brain injury radiographic phenotypes.

Do Students Effectively Regulate Their Use of Self-Testing as a Function of Item Difficulty?

Retrieval practice is beneficial for both easy-to-learn and difficult-to-learn materials, but scant research has examined students’ use of self-testing for items of varying difficulty. In two experiments, we investigated whether students differentially regulate their use of self-testing for easy and difficult items and assessed the effectiveness of students’ self-regulated choices. Undergraduate participants learned normatively easy and normatively difficult Lithuanian-English word pair translations. After an initial study trial, participants in the self-regulated learning groups chose whether they wanted to restudy an item, take a practice test, or remove an item from further practice. Participants chose to test items repeatedly while learning but dropped both easy and difficult items after reaching a criterion of about one correct recall per item. Consequently, final test performance 2 days later was lower for difficult items versus easy items, and performance was lower in the self-regulated learning group than in an experimenter-controlled comparison group (in Experiment 1). In Experiment 2, we tested hypotheses for why participants reached a similar number of correct recalls for both easy and difficult items. Three new groups included different scaffolds aimed at minimizing potential barriers to effective regulation. These scaffolds did not change participants’ learning choices, and as a result, performance on difficult items was still lower than on easy items. Importantly, participants planned to continue practicing items beyond one correct recall and believed that an optimal student should practice difficult items more than easy items, but they did not execute this plan during the learning task.

The effect of hint strength on the benefits of retrieval practice

AbstractRecent work has shown that retrieval hints can make test trials more enjoyable without sacrificing learning. We investigated the extent to which this effect was moderated by hint strength. Both experiments utilized a within‐participants design. In Experiment 1 (n = 41), participants studied skeletal charts highlighting a bone region. After study, participants received one test trial with either consonants‐only (e.g., _cc_p_t_l), vowels‐only (e.g., o__i_i_a_), or no hint (e.g., _________). Retrieval latencies were fastest for the consonants‐only items, suggesting they provided the strongest hints. Although the consonants‐only trial produced the worst final test performance, participants rated that trial as the most effective, the most fun, and the trial they would use from now on. Experiment 2 (n = 32) replicated and extended these results using a criterion learning paradigm. These experiments show that participants prefer extremely strong hints during test trials, even when such hints impair performance.

Development of a Simulation Surgical Cricothyroidotomy Curriculum for Novice Providers: A Learning Curve Study.

Airway obstruction is the third most common cause of preventable death on the battlefield, accounting for 1%-2% of total combat fatalities. No previous surgical cricothyroidotomy (SC) studies have analyzed the learning curve required to obtain proficiency despite being studied in numerous other surgical technique training experiments. The aims of this study were to establish expert SC performance criteria, develop a novel standardized SC curriculum, and determine the necessary number of practice iterations required by a novice to reach this pre-determined performance goal. A standardized checklist and SC performance standards were established based on the performance of 12 board certified Military Health System surgeons with prior experience on performing a SC using a simulated trauma mannequin. Expert-level criteria were defined as a SC time to completion of 40 s or less and checklist score of at least 9/10, including all critical steps. Study subjects included 89 novice providers (54 active-duty first- and second-year medical students and 35 Navy corpsmen). Subjects received instruction on performing a SC using the principles of mastery learning and performed a final test of SC proficiency on a trauma mannequin within a realistic simulated MEDEVAC helicopter. The total number of subject practice attempts, checklist scores, and time to completion were measured and/or blindly scored. Learning curve and exponential plateau equations were used to characterize their improvement in mean time to SC completion and checklist scores. Mean pre-test knowledge scores for the entire group were 11.8 ± 3.1 out of 24 points. Total mean practice learning plateaued at checklist scores of 9.9/10 after 7 iterations and at a mean completion time of 30.4 s after 10 iterations. During the final test performance in the helicopter, 67.4% of subjects achieved expert-level performance on the first attempt. All subjects achieved expert-level performance by the end of two additional attempts. While a significantly larger proportion of medical students (79.9%) successfully completed the helicopter test on the first attempt compared to corpsmen (54.3%), there were no statistically significant differences in mean SC completion times and checklist scores between both groups (P > 0.05). Medical students performed a SC only 1.3 s faster and scored only 0.16 points higher than corpsmen. The effect size for differences were small to negligible (Cohen's d range 0.18-0.33 for SC completion time; Cohen's d range 0.45-0.46 for checklist scores). This study successfully defined SC checklist scores and completion times based on the performance of experienced surgeons on a simulator. Using these criteria and the principles of mastery learning, novices with little knowledge and experience in SC were successfully trained to the level of experienced providers. All subjects met performance targets after training and overall performance plateaued after approximately seven iterations. Over two-thirds of subjects achieved the performance target on the first testing attempt in a simulated helicopter environment. Performance was comparable between medical student and corpsmen subgroups. Further research will assess the durability of maintaining SC skills and the timing for introducing refresher courses after initial skill acquisition.

Critical minds in Arabic and Islamic studies courses: A whiff of change

AbstractIn the present manuscript, we report the findings of a study that surveyed female college students' views of revamped Islamic and Arabic Culture courses embedded in the curriculum of a secular academic program in the Kingdom of Saudi Arabia. Of interest were students' views of and responses to recently implemented features of the curriculum and instruction that exemplified active learning. The ancillary aim was to determine whether views and responses were linked to performance. At the end of the semester, students recognized that the courses' curriculum and instruction fostered active learning opportunities, and expressed an appreciation for learning that is impactful. Results involving performance were mixed though. In fact, whereas recitation and final test performance decreased with students' recognition that instruction prioritized understanding of facts and ideas, greater discussion of matters that engaged students' attention, such as cognitive distortions and personal views, was associated with higher midterm and final test performance. It was concluded that the novelty of activities capable of engaging students in the learning process might explain students' appreciation of active learning as well as such activities' mixed relationship with performance outcomes.

Turn-by-turn Route Guidance Does Not Impair Route Learning

Turn-by-turn GPS guidance is useful when the navigator is uncertain about the correct route. Although route guidance is convenient, it comes at a spatial cognitive cost. Compared to unguided navigation, route guidance leads to poorer knowledge of the traversed environment. However, past research has not tested the effects of route guidance on route retracing, which is an important learning goal in many situations. Participants drove a pre-defined route in a driving simulator. All participants initially followed turn-by-turn directions twice (Experiment 1) or once (Experiment 2). Those in the Study condition continued to follow route guidance during two subsequent traversals, whereas those in the Test condition relied on memory and received corrective feedback. Following a 48-hr delay, participants completed a final test in which they retraced the route without guidance. Learning condition did not influence final test performance, indicating that route knowledge is unaffected by repeatedly following route guidance.

Investigating Test-Taking Strategies in Listening Assessment: A Comparative Study of Eye-Tracking and Self-Report Questionnaires

ABSTRACT This study aimed to investigate the test-taking strategies needed for successful completion of a lecture-based listening test by employing self-reported test-taking strategy use, actual strategy use measured via eye-tracking, and test scores. In this study, participants’ gaze behavior (measured by fixation and visit duration and frequency) were recorded while they completed two listening tests of three stages each: pre-listening, in which participants (n = 66) previewed question stems; while-listening, in which participants simultaneously listened to the recording and filled in their answers; and post-listening, in which they had time to review their answers and make necessary amendments. Following the listening tests, participants filled up a posttest questionnaire that asked about their strategy use in each of the three stages. Rasch measurement, t-test, and path analysis were performed on test scores, questionnaire results, and gaze patterns. Results suggest that gaze measures (visit duration and fixation frequency) predicted participants’ final test performance, while self-reports had moderate predicting power. The findings of this study have implications for the cognitive validity of listening tests, listening test design and pedagogical approaches in building listening competence.

Correlation Coefficient-Based Recombinative Guidance for Genetic Programming Hyperheuristics in Dynamic Flexible Job Shop Scheduling

Dynamic flexible job shop scheduling (JSS) is a challenging combinatorial optimization problem due to its complex environment. In this problem, machine assignment and operation sequencing decisions need to be made simultaneously under the dynamic environments. Genetic programming (GP), as a hyperheuristic approach, has been successfully used to evolve scheduling heuristics for dynamic flexible JSS. However, in traditional GP, recombination between parents may disrupt the beneficial building blocks by choosing the crossover points randomly. This article proposes a recombinative mechanism to provide guidance for GP to realize effective and adaptive recombination for parents to produce offspring. Specifically, we define a novel measure for the importance of each subtree of an individual, and the importance information is utilized to decide the crossover points. The proposed recombinative guidance mechanism attempts to improve the quality of offspring by preserving the promising building blocks of one parent and incorporating good building blocks from the other. The proposed algorithm is examined on six scenarios with different configurations. The results show that the proposed algorithm significantly outperforms the state-of-the-art algorithms on most tested scenarios, in terms of both final test performance and convergence speed. In addition, the rules obtained by the proposed algorithm have good interpretability.

Restudying With the Quiz in Hand: When Correct-Answer Feedback is No Better Than Minimal Feedback

Quizzing (testing) can enhance students’ metacognitive accuracy, restudy behaviors, and final test performance. Researchers have not examined, however, the effect of quiz feedback and the availability of the quiz during restudy on these indirect effects of quizzing. We examined three types of quiz feedback: correct-answer, right/wrong, and minimal. In Experiment 1, when the quiz and feedback were available during restudy, participants in all three conditions followed a discrepancy reduction study policy. There were no differences in final test performance in Experiment 1, implying that the degree of feedback elaboration did not impact learners’ benefit from quizzing and restudy. In Experiment 2, when we retracted access to the quiz and feedback during restudy, restudy was impaired, and the minimal condition performed worse on the final test than the correct-answer condition. It appears that correct-answer feedback is critical if quizzes are unavailable; if they are available, the type of feedback has much less impact.

Causal Attribution Preferences and Prospective Self-Assessment: The Unknowns of the Middle Eastern Learner

In the present field experiment, we examined the effects of a self-assessment exercise conducted in the middle of the semester on metacognitive awareness (i.e., the accuracy of self-assessment and its subjective confidence) and final test performance of college students of Middle Eastern descent. Effects were measured in the classroom against a business-as-usual control condition. It was hypothesized that if the exercise focuses students’ attention on internal causes (e.g., effort) in response to specific task demands, metacognitive awareness, metacognitive control, and, ultimately, final test performance would be enhanced. In poor performers, the exercise indeed improved the accuracy of self-assessment (as measured by grade estimates) and final performance. There was no evidence of the illusion of knowing phenomenon (i.e., the unrealistic belief that one knows or is able to perform a given task) among poor-performing students as their confidence in estimates remained low. Furthermore, the exercise did not change students’ causal attribution preferences, thereby suggesting that other dynamics are responsible for the effects of the self-assessment exercise.

The effect of practice test modality on perceived mental effort and delayed final test performance

ABSTRACT Taking a test on studied materials results in better delayed recall performance than restudying (a.k.a. the testing effect). A common finding in testing effect research is that the effect depends on test format: the magnitude of the testing effect differs between free-recall, cued-recall, and recognition testing. This is explained by the effortful retrieval hypothesis: effortful successful retrieval results in better memory for an item than less effortful successful retrieval. However, the assumption that successful retrieval on different types of tests requires different levels of effort has not yet been tested. To test this assumption, we measured perceived mental effort on different test formats. Participants indicated free-recall was more effortful than cued-recall, and cued-recall more effortful than recognition. Furthermore, cued and free-recall yielded better cued-recall performance on a one-week delayed test than restudy or recognition. The results support the assumption that different practice test formats require different levels of mental effort.

Beyond right or wrong: More effective feedback for formative multiple-choice tests.

IntroductionThe role of feedback in test-enhanced learning is an understudied area that has the potential to improve student learning. This study investigates the influence of different forms of post-test feedback on retention and transfer of biomedical knowledge within a test-enhanced learning framework.Methods64 participants from a Canadian and an Australian medical school sat two single-best-answer formative multiple choice tests one week apart. We compared the effects of conceptually focused, response-oriented, and simple right/wrong feedback on a learner’s ability to correctly answer new (transfer) questions. On the first test occasion, participants received parent items with feedback, and then attempted items closely related (near transfer) to and more distant (far transfer) from parent items. In a repeat test at 1 week, participants were given different near and far transfer versions of parent items. Feedback type, and near and far transfer items were randomized within and across participants.ResultsAnalysis demonstrated that response-oriented and conceptually focused feedback were superior to traditional right/wrong feedback for both types of transfer tasks and in both immediate and final retention test performance. However, there was no statistically significant difference between response-orientated and conceptually focused groups on near or far transfer problems, nor any differences in performance between our initial test occasion and the retention test 1 week later. As with most studies of transfer, participants’ far transfer scores were lower than for near transfer.DiscussionRight/wrong feedback appears to have limited potential to augment test-enhanced learning. Our work suggests that item-level feedback and feedback that identifies and elaborates on key conceptual knowledge are two important areas for future research on learning, retention and transfer.

A Close-Up on a Predictive Moment: Illusion of Knowing or Lack of Confidence in Self-Assessment?

Prediction allows learners to adjust behavior toward the future by exploiting information pertaining to the present and the past. Through a field study, we examined whether poor performing students are truly unaware of not knowing their deficiencies as the illusion of knowing (IoK) phenomenon implies. College students’ ability to predict their final test performance was surveyed as a function of experience (before and after the test), performance level, and self-efficacy. In this study, high performers’ prospective and retrospective predictions were more accurate and confident than those of poor performers. Although poor performers overestimated their grades (as predicted by IoK), they were less confident in their predictions. Furthermore, both their prediction accuracy and confidence benefited from the experience of taking the test. These findings, coupled with the lower self-efficacy of poor performers, suggest that prediction errors involving inflated estimations reflect the wishful thinking exhibited by students who are aware of their lack of competence but may have little confidence in their abilities.

Prompting retrieval during monitoring and self-regulated learning in older and younger adults

We evaluated age differences in the relationship between judgments of learning (JOLs) and the choice to restudy a subset of items under two conditions: 1) when a retrieval attempt was explicitly prompted during monitoring; 2) when a retrieval attempt was not explicitly prompted. Young and older adults studied unrelated word pairs. Item-by-item cue-only judgments followed, where participants either attempted to recall the target before providing a JOL or only provided a JOL. After the monitoring phase, participants reported how many total items they wanted to restudy. However, during the selection phase, participants selected half of the presented items to restudy. After restudying selected items, participants received a final cued recall test. Requiring individuals to attempt retrieval increased monitoring reaction times (RT) and decreased JOL magnitude, but did not affect self-regulated learning. For both monitoring groups, individuals were more likely to select items they rated with lower JOLs and items that they spent more time monitoring (i.e., greater RTs). In addition, older adults demonstrated a weaker negative relationship between JOLs and restudy selections, but no difference in the relationship between RTs and restudy selections, compared to young adults. Older adults also indicated wanting to restudy more total items than younger adults. Explicitly prompting retrieval during monitoring did not impact these observed age effects, or interestingly, final test performance. Overall, the results of this study suggest that prompting explicit retrieval prior to monitoring may have little direct effect on self-regulated learning or final test performance.