Intensive Longitudinal Assessment Research Articles

Intensive longitudinal assessment following index trauma to predict development of PTSD using machine learning

There are significant challenges to identifying which individuals require intervention following exposure to trauma, and a need for strategies to identify and provide individuals at risk for developing PTSD with timely interventions. The present study seeks to identify a minimal set of trauma-related symptoms, assessed during the weeks following traumatic exposure, that can accurately predict PTSD. Participants were 2185 adults (Mean age=36.4 years; 64% women; 50% Black) presenting for emergency care following traumatic exposure. Participants received a ‘flash survey’ with 6–8 varying symptoms (from a pool of 26 trauma symptoms) several times per week for eight weeks following the trauma exposure (each symptom assessed ∼6 times). Features (mean, sd, last, worst, peak-end scores) from the repeatedly assessed symptoms were included as candidate variables in a CART machine learning analysis to develop a pragmatic predictive algorithm. PTSD (PCL-5 ≥38) was present for 669 (31%) participants at the 8-week follow-up. A classification tree with three splits, based on mean scores of nervousness, rehashing, and fatigue, predicted PTSD with an Area Under the Curve of 0.836. Findings suggest feasibility for a 3-item assessment protocol, delivered once per week, following traumatic exposure to assess and potentially facilitate follow-up care for those at risk.

Intensive Longitudinal Adaptive Assessment: Item Selection and Stopping Rules in Highly Multidimensional Computerized Adaptive Tests

Intensive Longitudinal Adaptive Assessment: Item Selection and Stopping Rules in Highly Multidimensional Computerized Adaptive Tests

Intensive longitudinal assessment of mobility, social activity and loneliness in individuals with severe mental illness during COVID-19

Intensive longitudinal assessment of mobility, social activity and loneliness in individuals with severe mental illness during COVID-19

Patients’ well-being during the transition period after psychiatric hospitalization to school: insights from an intensive longitudinal assessment of patient–parent–teacher triads

BackgroundThe transition period after psychiatric hospitalization back to school is accompanied by various challenges, including a substantial risk for rehospitalization. Self-efficacy and self-control, as transdiagnostic variables and important predictors of coping with school demands, should be crucial factors for successful adaptation processes as well as an overall high well-being during school reentry. The present study therefore investigates how patients’ well-being develops during this period, and how it is related to patients’ self-control and academic self-efficacy, as well as parents’ and teachers’ self-efficacy in dealing with the patient.MethodsIn an intensive longitudinal design, daily ambulatory assessment measures via smartphone were collected with self-reports from the triadic perspective of 25 patients (Mage = 10.58 years), 24 parents, and 20 teachers on 50 consecutive school days, starting 2 weeks before discharge from a psychiatric day hospital (mean compliance rate: 71% for patients, 72% for parents and 43% for teachers). Patients answered daily questions between five and nine o'clock in the evening about their well-being, self-control, academic self-efficacy and about positive and negative events at school, as well as parents and teachers about their self-efficacy in dealing with the patient.ResultsMultilevel modeling revealed that on average, patients’ well-being and self-control decreased during the transition period, with trends over time differing significantly between patients. While patients’ academic self-efficacy did not systematically decrease over time, it did show considerable intra-individual fluctuation. Importantly, patients experienced higher well-being on days with higher self-control and academic self-efficacy as well as with higher parental self-efficacy. Daily teacher self-efficacy did not show a significant within-person relationship to daily patients’ well-being.ConclusionsWell-being in the transition period is related to self-control and self-efficacy of patients and their parents. Thus, addressing patients’ self-control and academic self-efficacy, as well as parental self-efficacy, seems promising to enhance and stabilize well-being of patients during transition after psychiatric hospitalization.Trial registration Not applicable, as no health care intervention was conducted.

Intensive Longitudinal Assessment of Adolescents to Predict Suicidal Thoughts and Behaviors

Suicide is a leading cause of death among adolescents. However, there are no clinical tools to detect proximal risk for suicide. Participants included 13- to 18-year-old adolescents (N= 103) reporting a current depressive, anxiety, and/or substance use disorder who owned a smartphone; 62% reported current suicidal ideation, with 25% indicating a past-year attempt. At baseline, participants were administered clinical interviews to assess lifetime disorders and suicidal thoughts and behaviors (STBs). Self-reports assessing symptoms and suicide risk factors also were obtained. In addition, the Effortless Assessment of Risk States (EARS) app was installed on adolescent smartphones to acquire daily mood and weekly suicidal ideation severity during the 6-month follow-up period. Adolescents completed STB and psychiatric service use interviews at the 1-, 3-, and 6-month follow-up assessments. K-means clustering based on aggregates of weekly suicidal ideation scores resulted in a 3-group solution reflecting high-risk (n= 26), medium-risk (n= 47), and low-risk (n= 30) groups. Of the high-risk group, 58% reported suicidal events (ie, suicide attempts, psychiatric hospitalizations, emergency department visits, ideation severity requiring an intervention) during the 6-month follow-up period. For participants in the high-risk and medium-risk groups (n = 73), mood disturbances in the preceding 7 days predicted clinically significant ideation, with a 1-SD decrease in mood doubling participants' likelihood of reporting clinically significant ideation on a given week. Intensive longitudinal assessment through use of personal smartphones offers a feasible method to assess variability in adolescents' emotional experiences and suicide risk. Translating these tools into clinical practice may help to reduce the needless loss of life among adolescents.

An Intensive Longitudinal Assessment Approach to Surveilling Trajectories of Burnout over the First Year of the COVID Pandemic.

Frontline clinicians responding to the COVID-19 pandemic are at increased risk of burnout, but less is known about the trajectory of clinician burnout as caseloads increase and decrease. Personal and professional resources, including self-efficacy and hospital support, can attenuate the risk of burnout. Yet, empirical data documenting how burnout and resources changed as the pandemic waxed and waned are limited. This intensive longitudinal prospective study employed ecological momentary assessment methods to examine trajectories of burnout and resources over the pandemic's first year in a New York City hospital. A 10-item survey was emailed every 5 days to frontline clinicians (physicians, nurses, and physician assistants). The primary outcome was a single-item validated measure of burnout; predictors included daily hospital COVID-19-related caseloads and personal and professional resources. Clinicians (n = 398) completed the initial survey and an average of 12 surveys over the year. Initially, 45.3% of staff reported burnout; over the year, 58.7% reported burnout. Following the initial COVID peak, caseloads declined, and burnout levels declined. During the second wave of COVID, as caseloads increased and remained elevated and personal and professional resource levels decreased, burnout increased. This novel application of intensive longitudinal assessment enabled ongoing surveillance of burnout and permitted us to evaluate how fluctuations in caseload intensity and personal and professional resources related to burnout over time. The surveillance data support the need for intensified resource allocation during prolonged pandemics.

Ecological Momentary Assessment: A Systematic Review of Validity Research.

Ecological momentary assessment (EMA) is a self-report method that involves intensive longitudinal assessment of behavior and environmental conditions during everyday activities. EMA has been used extensively in health and clinical psychology to investigate a variety of health behaviors, including substance use, eating, medication adherence, sleep, and physical activity. However, it has not been widely implemented in behavior analytic research. This is likely an example of the empirically based skepticism with which behavioral scientists view self-report measures. We reviewed studies comparing electronic, mobile EMA (mEMA) to more objective measures of health behavior to explore the validity of mEMA as a measurement tool, and to identify procedures and factors that may promote the accuracy of mEMA. We identified 32 studies that compared mEMA to more objective measures of health behavior or environmental events (e.g., biochemical measures or automated devices such as accelerometers). Results showed that the correspondence rates varied considerably across individuals, behavior, and studies (agreement rates ranged from 1.8%-100%), and no unifying variables could be identified across the studies that found high correspondence. The findings suggest that mEMA can be an accurate measurement tool, but further research should be conducted to identify procedures and variables that promote accurate responding.

Developing an item pool to assess processes of change in psychological interventions: The Process-Based Assessment Tool (PBAT)

Process-based therapy (PBT) focuses on treatment elements that target biopsychosocial processes of relevance to individual treatment goals. This focus requires new, more integrative and idionomic models that identify key processes of change, using high temporal density measurement applied at the level of the person. Standard measurement validation approaches are inadequate to this challenge. The present study develops and provides a preliminary validation of a process-based assessment tool (PBAT) -- an item pool meant for intensive longitudinal clinical assessment. Developed using the Extended-Evolutionary Meta-Model of PBT and evaluated using a machine-learning algorithm appropriate for the evaluation of individual items, we administered the PBAT online to a sample of 598 participants (290 male; 302 female; 6 unidentified. Mage = 32.6). Analyses revealed that the PBAT distinguishes between positive and negative processes, links in theoretically coherent ways to need satisfaction and thwarting, and links to clinically relevant outcomes of sadness, anger, anxiety, stress, lack of social support, vitality, and health. The PBAT provides a beginning step towards developing a process-based tool that allows clinicians and researchers to select individual items or sets of items for individual-focused idionomic research and practice.

Intensive Longitudinal Assessment of College Transition and Psychiatric Illness Supports an Energy Allostatic Model of Depression and Sleep-Wake Dynamics

The transition to adulthood and periods of depression both impact sleep-wake behavior transitions. Thus, while sleep-wake behavior is broadly homeostatic the system has buffering capacity to recover from deficits under adaptive conditions. Under unexpected risk landscapes or in response to chronic stressors, individuals may adapt sleep-wake behavior patterns that are locally favorable, but fail to meet broader homeostatic requirements.

Networks of stress, affect and eating behaviour: anticipated stress coping predicts goal-congruent eating in young adults

BackgroundMany people aim to eat healthily. Yet, affluent food environments encourage consumption of energy dense and nutrient-poor foods, making it difficult to accomplish individual goals such as maintaining a healthy diet and weight. Moreover, goal-congruent eating might be influenced by affects, stress and intense food cravings and might also impinge on these in turn. Directionality and interrelations of these variables are currently unclear, which impedes targeted intervention. Psychological network models offer an exploratory approach that might be helpful to identify unique associations between numerous variables as well as their directionality when based on longitudinal time-series data.MethodsAcross 14 days, 84 diet-interested participants (age range: 18–38 years, 85.7% female, mostly recruited via universities) reported their momentary states as well as retrospective eating episodes four times a day. We used multilevel vector autoregressive network models based on ecological momentary assessment data of momentary affects, perceived stress and stress coping, hunger, food craving as well as goal-congruent eating behaviour.ResultsNeither of the momentary measures of stress (experience of stress or stress coping), momentary affects or craving uniquely predicted goal-congruent eating. Yet, temporal effects indicated that higher anticipated stress coping predicted subsequent goal-congruent eating. Thus, the more confident participants were in their coping with upcoming challenges, the more they ate in line with their goals.ConclusionMost eating behaviour interventions focus on hunger and craving alongside negative and positive affect, thereby overlooking additional important variables like stress coping. Furthermore, self-regulation of eating behaviours seems to be represented by how much someone perceives a particular eating episode as matching their individual eating goal. To conclude, stress coping might be a potential novel intervention target for eating related Just-In-Time Adaptive Interventions in the context of intensive longitudinal assessment.

Capturing Fluctuations in Gendered Cognition With Novel Intensive Longitudinal Measures

Cognition is often assumed to be stable, but compelling evidence shows that some skills (e.g., working memory) vary. There is limited investigation, however, of variation in gendered cognition, such as spatial skills and verbal recall, potentially because there are no validated measures for intensive longitudinal assessment. The goal of this study was to introduce and validate new 75-occasion measures of three-dimensional mental rotations and delayed paired verbal recall. This was accomplished by studying cognitive fluctuations over 75 days in an intensive longitudinal study of 121 participants, focusing on 54 (27 women) who were matched for gender, age, and language. Results from psychometric analyses suggest that both measures show parallel forms reliability, including expected gender differences, and validity (i.e., performance with respect to a standard measure or item difficulty), although results are more consistent for mental rotations than verbal recall. Results also suggest there are significant daily fluctuations in both mental rotations and verbal recall. Findings encourage future clinical work on the antecedents and consequences of cognitive fluctuations with the newly developed, freely available measures.

Comparability of Emotion Dynamics Derived From Ecological Momentary Assessments, Daily Diaries, and the Day Reconstruction Method: Observational Study.

BackgroundInterest in the measurement of the temporal dynamics of people’s emotional lives has risen substantially in psychological and medical research. Emotions fluctuate and change over time, and measuring the ebb and flow of people’s affective experiences promises enhanced insights into people’s health and functioning. Researchers have used a variety of intensive longitudinal assessment (ILA) methods to create measures of emotion dynamics, including ecological momentary assessments (EMAs), end-of-day (EOD) diaries, and the day reconstruction method (DRM). To date, it is unclear whether they can be used interchangeably or whether ostensibly similar emotion dynamics captured by the methods differ in meaningful ways.ObjectiveThis study aims to examine the extent to which different ILA methods yield comparable measures of intraindividual emotion dynamics.MethodsData from 90 participants aged 50 years or older were collected in a probability-based internet panel, the Understanding America Study, and analyzed. Participants provided positive and negative affect ratings using 3 ILA methods: (1) smartphone-based EMA, administered 6 times per day over 1 week, (2) web-based EOD diaries, administered daily over the same week, and (3) web-based DRM, administered once during that week. We calculated 11 measures of emotion dynamics (addressing mean levels, variability, instability, and inertia separately for positive and negative affect, as well as emotion network density, mixed emotions, and emotional dialecticism) from each ILA method. The analyses examined mean differences and correlations of scores addressing the same emotion dynamic across the ILA methods. We also compared the patterns of intercorrelations among the emotion dynamics and their relationships with health outcomes (general health, pain, and fatigue) across ILA methods.ResultsEmotion dynamics derived from EMAs and EOD diaries demonstrated moderate-to-high correspondence for measures of mean emotion levels (ρ≥0.95), variability (ρ≥0.68), instability (ρ≥0.51), mixed emotions (ρ=0.92), and emotional dialecticism (ρ=0.57), and low correspondence for measures of inertia (ρ≥0.17) and emotion network density (ρ=0.36). DRM-derived measures showed correlations with EMAs and EOD diaries that were high for mean emotion levels and mixed emotions (ρ≥0.74), moderate for variability (ρ=0.38-.054), and low to moderate for other measures (ρ=0.03-0.41). Intercorrelations among the emotion dynamics showed high convergence across EMAs and EOD diaries, and moderate convergence between the DRM and EMAs as well as EOD diaries. Emotion dynamics from all 3 ILA methods produced very similar patterns of relationships with health outcomes.ConclusionsEMAs and EOD diaries provide corresponding information about individual differences in various emotion dynamics, whereas the DRM provides corresponding information about emotion levels and (to a lesser extent) variability, but not about more complex emotion dynamics. Our results caution researchers against viewing these ILA methods as universally interchangeable.

Understanding Personality through Patterns of Daily Socializing: Applying Recurrence Quantification Analysis to Naturalistically Observed Intensive Longitudinal Social Interaction Data

Ambulatory assessment methods provide a rich approach for studying daily behaviour. Too often, however, these data are analysed in terms of averages, neglecting patterning of this behaviour over time. This paper describes recurrence quantification analysis (RQA), a non–linear time series technique for analysing dynamic systems, as a method for analysing patterns of categorical, intensive longitudinal ambulatory assessment data. We apply RQA to objectively assessed social behaviour (e.g. talking to another person) coded from the Electronically Activated Recorder. Conceptual interpretations of RQA parameters, and an analysis of Electronically Activated Recorder data in adults going through a marital separation, are provided. Using machine learning techniques to avoid model overfitting, we find that adding RQA parameters to models that include just average amount of time spent talking (a static measure) improves prediction of four Big Five personality traits: extraversion, neuroticism, conscientiousness, and openness. Our strongest results suggest that a combination of average amount of time spent talking and four RQA parameters yield an R2 = .09 for neuroticism. Neuroticism is shown to be associated with shorter periods of extended conversation (periods of at least 12 minutes), demonstrating the utility of RQA to identify new relationships between personality and patterns of daily behaviour. Materials: https://osf.io/5nkr9/ . © 2020 European Association of Personality Psychology

Associations between mindfulness and mental health outcomes: A systematic review of ecological momentary assessment research.

Psychological science has taken up investigations of the effectiveness of mindfulness-based programs (MBPs) and mechanisms through which people benefit from mindfulness. Reliable and valid psychometric tools are essential components of psychological science, and efforts have been made to produce tools for the accurate measurement of mindfulness as a construct. However, trait measurement methods, which are commonly used, may not adequately assess mindfulness and mental health outcomes in a way that allows for mechanisms to be adequately tested. Intensive longitudinal assessment methods sample behavior and experience multiple times over a brief period of several days or weeks, and may be more appropriate methods for testing mechanisms of action. We provide a systematic review of published, peer-reviewed studies that used intensive longitudinal methods to investigate the effects of mindfulness on mental health outcomes. Articles were included in the systematic review if mindfulness measures and/or mindfulness interventions were a part of the study design and if intensive longitudinal methods were used to assess mindfulness or mental health outcomes. Findings consistently demonstrated a positive association between mindfulness and mental health. Only two studies collected both trait and state measurements of either mindfulness or mental health outcomes, and results indicated that EMA produced larger effect sizes between mindfulness and mental health outcomes. Theorized associations between mindfulness and mental health are supported by the current EMA literature. Intensive longitudinal methods may produce more consistent and reliable results through increased sensitivity and ecological validity in that they examine the momentary relationships between mindfulness and mental health outcomes. Thus, intensive longitudinal assessment may be a more appropriate method for investigating hypothesized mechanisms of action in MBPs.

Quality of life in people with dementia living in nursing homes: validation of an eight-item version of the QUALIDEM for intensive longitudinal assessment

PurposeOur aim was to examine whether quality of life which was repeatedly assessed over time is related with the comprehensive assessment of quality of life (QoL) and thereby to validate a brief QoL assessment.MethodThis longitudinal study used a comprehensive assessment of quality of life at baseline (QUALIDEM; 37 items) to validate an eight-item version of QUALIDEM to assess momentary quality of life which was repeatedly administered using a tablet device after baseline. In all, 150 people with dementia from 10 long-term facilities participated. Momentary quality of life and comprehensive quality of life, age, gender, activities of daily living (Barthel Index), Functional assessment staging (FAST), and Geriatric Depression (GDS) have been assessed.ResultsComprehensive and momentary quality of life showed good internal consistency with Cronbach’s alpha of .86 and .88 to .93, respectively. For multiple associations of momentary quality of life with the comprehensive quality of life, momentary quality of life was significantly related to comprehensive quality of life (B = .14, CI .08/.20) and GDS (B = − .13, CI − .19/− .06). More specifically, the comprehensive QUALIDEM subscales ‘positive affect’, ‘negative affect’, ‘restlessness’, and ‘social relationships’ showed significant positive associations with momentary quality of life (p < .001).ConclusionWe found that momentary quality of life, reliably assessed by tablet, was associated with comprehensive measures of quality of life and depressive symptoms in people with dementia. Broader use of tablet-based assessments within frequent QoL measurements may enhance time management of nursing staff and may improve the care quality and communication between staff and people with dementia.

Annual Research Review: Adolescent mental health in the digital age: facts, fears, and future directions.

Adolescents are spending an increasing amount of their time online and connected to each other via digital technologies. Mobile device ownership and social media usage have reached unprecedented levels, and concerns have been raised that this constant connectivity is harming adolescents' mental health. This review synthesized data from three sources: (a) narrative reviews and meta-analyses conducted between 2014 and 2019, (b) large-scale preregistered cohort studies and (c) intensive longitudinal and ecological momentary assessment studies, to summarize what is known about linkages between digital technology usage and adolescent mental health, with a specific focus on depression and anxiety. The review highlights that most research to date has been correlational, focused on adults versus adolescents, and has generated a mix of often conflicting small positive, negative and null associations. The most recent and rigorous large-scale preregistered studies report small associations between the amount of daily digital technology usage and adolescents' well-being that do not offer a way of distinguishing cause from effect and, as estimated, are unlikely to be of clinical or practical significance. Implications for improving future research and for supporting adolescents' mental health in the digital age are discussed.

Short-term prediction of suicidal thoughts and behaviors in adolescents: Can recent developments in technology and computational science provide a breakthrough?

BackgroundSuicide is one of the leading causes of death among adolescents, and developing effective methods to improve short-term prediction of suicidal thoughts and behaviors (STBs) is critical. Currently, the most robust predictors of STBs are demographic or clinical indicators that have relatively weak predictive value. However, there is an emerging literature on short-term prediction of suicide risk that has identified a number of promising candidates, including (but not limited to) rapid escalation of: (a) emotional distress, (b) social dysfunction (e.g., bullying, rejection), and (c) sleep disturbance. However, these prior studies are limited in two critical ways. First, they rely almost entirely on self-report. Second, most studies have not focused on assessment of these risk factors using intensive longitudinal assessment techniques that are able to capture the dynamics of changes in risk states at the individual level. MethodIn this paper we explore how to capitalize on recent developments in real-time monitoring methods and computational analysis in order to address these fundamental problems. ResultsWe now have the capacity to use: (a) smartphone, wearable computing, and smart home technology to conduct intensive longitudinal assessments monitoring of putative risk factors with minimal participant burden and (b) modern computational techniques to develop predictive algorithms for STBs. Current research and theory on short-term risk processes for STBs, combined with the emergent capabilities of new technologies, suggest that this is an important research agenda for the future. LimitationsAlthough these approaches have enormous potential to create new knowledge, the current empirical literature is limited. Moreover, passive monitoring of risk for STBs raises complex ethical issues that will need to be resolved before large scale clinical applications are feasible. ConclusionsSmartphone, wearable, and smart home technology may provide one point of access that might facilitate both early identification and intervention implementation, and thus, represents a key area for future STB research.

F242. Intensive Longitudinal Assessment of Mania and Psychosis Using Commonly Available Technologies

Individuals with affective and psychotic illnesses often experience profound changes in mood and cognition over time, and over relatively short periods (e.g., hospitalization). The nature of these fluctuations often vary significantly from individual to individual, posing tremendous challenges for clinical practice and group comparison studies. And yet, patterns often emerge within an individual, allowing clinicians to optimize treatment over longitudinal assessments.

The Links Between Social Anxiety Disorder, Insomnia Symptoms, and Alcohol Use Disorders: Findings From a Large Sample of Adolescents in the United States

Social anxiety disorder (SAD) is associated with increased risk of developing an alcohol use disorder (AUD). Most of the current literature has focused on the role of acute stress responding in this relation; however, both SAD and AUDs also are linked to insomnia symptoms (i.e., difficulty falling or staying asleep). As adolescence is a sensitive period for the onset of these disorders, the present study examined if insomnia symptoms might partially account for the SAD-AUD link in a large sample of adolescents. Data from the National Comorbidity Survey–Adolescent Supplement were examined. Participants (N = 10,140) completed interviews to assess past 12-month SAD and AUD diagnostic status as well as insomnia symptoms. Analyses tested whether insomnia symptoms accounted for a significant proportion of the SAD-AUD relation. Results indicated that insomnia symptoms were positively related to both SAD and AUD status, and the relation between SAD and AUD status was significantly reduced when insomnia symptoms were included in the model. Findings remained significant after controlling for the effects of age, gender, posttraumatic stress disorder, major depressive disorder, and other drug dependence status. Experimental examination and intensive longitudinal assessment of these relationships are needed before strong conclusions can be inferred about causality and temporal relationships. The current findings do indicate insomnia may be an important indirect and stigma-free treatment target to address in prevention and treatment efforts for SAD, AUDs, and their co-occurrence.

Association between Participation in an Intensive Longitudinal Assessment Program and Performance on a Cognitive Examination in the Maintenance of Certification in Anesthesiology Program®.

As part of the Maintenance of Certification in Anesthesiology Program® (MOCA®), the American Board of Anesthesiology (Raleigh, North Carolina) developed the MOCA Minute program, a web-based intensive longitudinal assessment involving weekly questions with immediate feedback and links to learning resources. This observational study tested the hypothesis that individuals who participate in the MOCA Minute program perform better on the MOCA Cognitive Examination (CE) compared with those who do not participate. Two separate cohorts of individuals eligible for July 2014 and January 2015 CEs were invited to participate in this pilot. The CE scores for each cohort were compared between those who did and did not participate, controlling for the factors known to affect performance. For the first cohort, examination performances for topics covered and not covered by the MOCA Minute were analyzed separately. Six hundred sixteen diplomates in July 2014 and 684 diplomates in January 2015 took the CE for the first time. In multiple regression analysis, those actively participating scored 9.9 points (95% CI, 0.8 to 18.9) and 9.3 points (95% CI, 2.3 to 16.3) higher when compared with those not enrolled, respectively. Compared to the group that did not enroll in MOCA Minute, those who enrolled but did not actively participate demonstrated no improvement in scores. MOCA Minute participation was associated with improvement in both questions covering topics included the MOCA Minute and questions not covering these topics. This analysis provides evidence that voluntary active participation in a program featuring frequent knowledge assessments accompanied by targeted learning resources is associated with improved performance on a high-stakes CE.