Hypothesis Resident readiness to perform procedures is germane to patient safety and quality of outcomes. We previously described a pragmatic assessment instrument for infant lumbar puncture (ILP) on which a high rating is predictive of clinical success.(1) This tool can help supervisors determine when an intern is ready to perform their first supervised ILP. We explored the implementation and impact of this just-in-time assessment on interns’ clinical success rates with ILP over several academic years. Methods This prospective study enrolled cohorts of interns in pediatric and emergency medicine over two years (2012-2014). All interns completed a simulation-based mastery learning session with individually coached deliberate practice until a predefined mastery performance standard was achieved. Intervention: When an intern had a patient requiring an ILP the supervisor conducted a just-in-time simulation-based assessment of their performance. Readiness to perform the clinical procedure (with supervision) was determined using a previously validated 4 point scale. Interns that did not achieve a passing score were not supposed to perform the clinical procedure. Comparison: Individuals that did not comply with the assessment pathway. Outcome measure: Interns self-reported on their first LP via on-line questionnaire (success was defined as obtaining cerebrospinal fluid with <1000 red blood cells or described as clear on the first needle insertion). Results Over two academic years, 1577 interns from 48 sites participated and reported a total of 753 procedures. The LP success rate for interns complying with the assessment pathway was 183/458 (40%) compared to 102/284 (36%) for those who did not comply with the pathway prior to their first LP attempt (Difference = 6%, 95% CI -3%, to 11%). 458/753(61%) of interns were compliant with the assessment pathway prior to their first LP. There was improvement in both pathway adherence and success rate between the first- and second-year of this new intervention (table 1). Conclusion Just-in-time simulation-based competency assessment is feasible in an academic setting. However, requiring interns to pass an assessment prior to performing the procedure did not produce a large clinical effect on success rates with ILP. Larger sample size is needed to detect smaller clinical effect sizes while concurrently working to improve compliance and cultural acceptance of a competency assessment pathway.