Response Process Validity Research Articles

Experts’ responses in script concordance tests: a response process validity investigation

The script concordance test (SCT), designed to measure clinical reasoning in complex cases, has recently been the subject of several critical research studies. Amongst other issues, response process validity evidence remains lacking. We explored the response processes of experts on an SCT scoring panel to better understand their seemingly divergent beliefs about how new clinical data alter the suitability of proposed actions within simulated patient cases. A total of 10 Argentine gastroenterologists who served as the expert panel on an existing SCT re-answered 15 cases 9 months after their original panel participation. They then answered questions probing their reasoning and reactions to other experts' perspectives. The experts sometimes noted they would not ordinarily consider the actions proposed for the cases at all (30/150 instances [20%]) or would collect additional data first (54/150 instances [36%]). Even when groups of experts agreed about how new clinical data in a case affected the suitability of a proposed action, there was often disagreement (118/133 instances [89%]) about the suitability of the proposed action before the new clinical data had been introduced. Experts reported confidence in their responses, but showed limited consistency with the responses they had given 9 months earlier (linear weighted kappa = 0.33). Qualitative analyses showed nuanced and complex reasons behind experts' responses, revealing, for example, that experts often considered the unique affordances and constraints of their varying local practice environments when responding. Experts generally found other experts' alternative responses moderately compelling (mean ± standard deviation 2.93 ± 0.80 on a 5-point scale, where 3 = moderately compelling). Experts switched their own preferred responses after seeing others' reasoning in 30 of 150 (20%) instances. Expert response processes were not consistent with the classical interpretation and use of SCT scores. However, several fruitful and justifiable alternatives for the use of SCT-like methods are proposed, such as to guide assessments for learning.

ACGME Milestones Within Subspecialty Training Programs: One Institution's Experience.

The Accreditation Council for Graduate Medical Education Milestones were created as a criterion-based framework to promote competency-based education during graduate medical education. Despite widespread implementation across subspecialty programs, extensive validity evidence supporting the use of milestones within fellowship training is lacking. We assessed the construct and response process validity of milestones in subspecialty fellowship programs in an academic medical center. From 2014-2016, we performed a single center retrospective cohort analysis of milestone data from fellows across 5 programs. We analyzed summary statistics and performed multivariable linear regression to assess change in milestone ratings by training year and variability in ratings across fellowship programs. Finally, we examined a subset of Professionalism and Interpersonal and Communication Skills subcompetencies from the first 6 months of training to identify the proportion of fellows deemed "ready for independent practice" in these domains. Milestone data were available for 68 fellows, with 75 933 unique subcompetency ratings. Multivariable linear regression, adjusted for subcompetency and subspecialty, revealed an increase of 0.17 (0.16-0.19) in ratings with each postgraduate year level increase (P < .005), as well as significant variation in milestone ratings across subspecialties. For the Professionalism and Interpersonal and Communication Skills domains, mean ratings within the first 6 months of training were 3.78 and 3.95, respectively. We noted a minimal upward trend of milestone ratings in subspecialty training programs, and significant variability in implementing milestones across differing subspecialties. This may suggest possible difficulties with the construct validity and response process of the milestone system in certain medical subspecialties.

Competency-based Standard Setting for a High-stakes Objective Structured Clinical Examination (OSCE): Validity Evidence.

This article was migrated. The article was marked as recommended. Introduction Medical educators need to demonstrate that their trainees meet expected competency levels when progressing through medical education. This study aimed to develop competency-based pass/fail cut-scores for a graduation required Objective Structured Clinical Examination (OSCE), and examine validity evidence for new standards. Methods Six clinicians used the modified Angoff method to determine the cut-scores for an 8-station OSCE. The clinicians estimated the percentage of minimally competent students who would answer each checklist item correctly. Inter-rater reliability, differences in other academic achievements between pass/fail groups, educational impact, and response process were examined. Results One hundred seventy-four rising 4th-year medical students participated in the OSCE. The cut-scores determined for the OSCE resulted in a substantially lower failure rate (5% vs. 29% of the previous year). The inter-rater reliability across domains and cases was .98 (95% CI = .97 - .99). The pass/fail groups significantly differed in six of the eight measures of academic achievements included in the study. Discussion The impact of the standards setting was substantial as it significantly reduced the failure rate and burdens of remediation for both students and faculty. The very high inter-rater reliability indicates that the modified Angoff method produced reliable cut-scores. The significant differences between the pass/fail groups in other measures support external validity of the standards and ensure no false passes. The study also supports response process validity by including discussion among judges and check of previous student performances, as well as recruiting and training multiple clinician educators experienced in medical student teaching. Conclusion Findings of the study provide strong evidence supporting validity of the new cut-scores from a wide spectrum of validity metrics, including response process, internal structure, relations to other variables, and consequences. The study also added to the literature the value of the modified Angoff method in determining competency-based standards for OSCEs.

Examining the effects of gaming and guessing on script concordance test scores.

IntroductionIn a script concordance test (SCT), examinees are asked to judge the effect of a new piece of clinical information on a proposed hypothesis. Answers are collected using a Likert-type scale (ranging from −2 to +2, with ‘0’ indicating no effect), and compared with those of a reference panel of ‘experts’. It has been argued, however, that SCT may be susceptible to the influences of gaming and guesswork. This study aims to address some of the mounting concern over the response process validity of SCT scores.MethodUsing published datasets from three independent SCTs, we investigated examinee response patterns, and computed the score a hypothetical examinee would obtain on each of the tests if he 1) guessed random answers and 2) deliberately answered ‘0’ on all test items.ResultsA simulated random guessing strategy led to scores 2 SDs below mean scores of actual respondents (Z-scores −3.6 to −2.1). A simulated ‘all-0’ strategy led to scores at least 1 SD above those obtained by random guessing (Z-scores −2.2 to −0.7). In one dataset, stepwise exclusion of items with modal panel response ‘0’ to fewer than 10% of the total number of test items yielded hypothetical scores 2 SDs below mean scores of actual respondents.DiscussionRandom guessing was not an advantageous response strategy. An ‘all-0’ response strategy, however, demonstrated evidence of artificial score inflation. Our findings pose a significant threat to the SCT’s validity argument. ‘Testwiseness’ is a potential hazard to all testing formats, and appropriate countermeasures must be established. We propose an approach that might be used to mitigate a potentially real and troubling phenomenon in script concordance testing. The impact of this approach on the content validity of SCTs merits further discussion.

Professional Identity Formation in medical school: One measure reflects changes during pre-clerkship training.

This article was migrated. The article was marked as recommended. Professional Identity Formation (PIF), the process of internalizing a profession's core values and beliefs, is an explicit goal of medical education. The Professional Identity Essay (PIE), a developmental measure of the extent to which individuals have a complex and self-defined understanding of their professional role, is a tool to both study and scaffold PIF. PIE staging has internal reliability and response process validity and correlates with a validated measure of moral reasoning. In this study, we investigate whether PIF, as measured by PIE, changes during pre-clerkship training. Medical students in the class of 2019 completed the PIE during orientation to medical school (PIE#1) and 15 months later, during orientation to clerkships (PIE#2), to the same prompts. These written responses are PIF-staged by an expert rater. On average, PIF scores reveal that 46% of the group remained at the same stage as they were on entry to medical school, 42% scored at a higher stage of PIF, and 15% of students scored at a lower stage of PIF after pre-clerkship training. This result suggests that medical students are heterogeneous with respect to the development of their medical PIF early in medical school training.

Development and validation of a Chinese translated questionnaire: Asingle simultaneous tool for assessing gastrointestinal and upper respiratory tract related illnesses in pre-school children.

ObjectivesChildren are prone to contagious illnesses that come from peers in nurseries, kindergartens, and day care centres. The administration of probiotics has been reported to decrease the episodes of such illnesses, leading to decreased absences and consumption of antibiotics. With less emphasis on, and preferences for, blood collection from young subjects, quantifiable data are merely obtained from surveys and questionnaires. Malaysia has a population which is 25% ethnic Chinese. We aimed to develop a single tool that enables simultaneous assessments of both gastrointestinal and respiratory tract-related illnesses among young Chinese children.MethodsThe English-language validated questionnaires using data about demographics and monthly health records were translated into the Chinese language. Both forward and backward translated versions were validated.ResultsThe developed demographic and monthly health questionnaires showed an overall item-level content validity index (I-CVI) of 0.99 and 0.97, respectively; while the translated Chinese versions showed I-CVI of 0.97 and 0.98, respectively. Item-level of response process validity index of 1.00 for this questionnaire was obtained from 30 respondents inferring that the items were clear and comprehensible.ConclusionsThis study showed acceptable levels validity in the Chinese translated version, illustrating a valid and reliable tool to be used for simultaneous assessment of gastrointestinal and respiratory tract-related illnesses in young children that is applicable for Malaysia's Chinese population and other Chinese-speaking nations.

Voluntary vs. compulsory student evaluation of clerkships: effect on validity and potential bias

BackgroundStudents evaluations of their learning experiences can provide a useful source of information about clerkship effectiveness in undergraduate medical education. However, low response rates in clerkship evaluation surveys remain an important limitation. This study examined the impact of increasing response rates using a compulsory approach on validity evidence.MethodsData included 192 responses obtained voluntarily from 49 third-year students in 2014–2015, and 171 responses obtained compulsorily from 49 students in the first six months of the consecutive year at one medical school in Lebanon. Evidence supporting internal structure and response process validity was compared between the two administration modalities. The authors also tested for potential bias introduced by the use of the compulsory approach by examining students’ responses to a sham item that was added to the last survey administration.ResultsResponse rates increased from 56% in the voluntary group to 100% in the compulsory group (P < 0.001). Students in both groups provided comparable clerkship rating except for one clerkship that received higher rating in the voluntary group (P = 0.02). Respondents in the voluntary group had higher academic performance compared to the compulsory group but this difference diminished when whole class grades were compared. Reliability of ratings was adequately high and comparable between the two consecutive years. Testing for non-response bias in the voluntary group showed that females were more frequent responders in two clerkships. Testing for authority-induced bias revealed that students might complete the evaluation randomly without attention to content.ConclusionsWhile increasing response rates is often a policy requirement aimed to improve the credibility of ratings, using authority to enforce responses may not increase reliability and can raise concerns over the meaningfulness of the evaluation. Administrators are urged to consider not only response rates, but also representativeness and quality of responses in administering evaluation surveys.

Validation of Multisource Feedback in Assessing Medical Performance: A Systematic Review.

Over the past 10 years, a number of systematic reviews have evaluated the validity of multisource feedback (MSF) to assess and quality-assure medical practice. The purpose of this study is to synthesize the results from existing reviews to provide a holistic overview of the validity evidence. This review identified eight systematic reviews evaluating the validity of MSF published between January 2006 and October 2016. Using a standardized data extraction form, two independent reviewers extracted study characteristics. A framework of validation developed by the American Psychological Association was used to appraise the validity evidence within each systematic review. In terms of validity evidence, each of the eight reviews demonstrated evidence across at least one domain of the American Psychological Association's validity framework. Evidence of assessment validity within the domains of "internal structure" and "relationship to other variables" has been well established. However, the domains of content validity (ie, ensuring that MSF tools measure what they are intended to measure); consequential validity (ie, evidence of the intended or unintended consequences MSF assessments may have on participants or wider society), and response process validity (ie, the process of standardization and quality control in the delivery and completion of assessments) remain limited. Evidence for the validity of MSF has, across a number of domains, been well established. However, the size and quality of the existing evidence remains variable. To determine the extent to which MSF is considered a valid instrument to assess medical performance, future research is required to determine the following: (1) how best to design and deliver MSF assessments that address the identified limitations of existing tools and (2) how to ensure that involvement within MSF supports positive changes in practice. Such research is integral if MSF is to continue to inform medical performance and subsequent improvements in the quality and safety of patient care.

Behaviors in Advance Care Planning and ACtions Survey (BACPACS): development and validation part 1

BackgroundAlthough advance care planning (ACP) is fairly well understood, significant barriers to patient participation remain. As a result, tools to assess patient behaviour are required. The objective of this study was to improve the measurement of patient engagement in ACP by detecting existing survey design issues and establishing content and response process validity for a new survey entitled Behaviours in Advance Care Planning and ACtions Survey (BACPACS).MethodsWe based our new tool on that of an existing ACP engagement survey. Initial item reduction was carried out using behavior change theories by content and design experts to help reduce response burden and clarify questions. Thirty-two patients with chronic diseases (cancer, heart failure or renal failure) were recruited for the think aloud cognitive interviewing with the new, shortened survey evaluating patient engagement with ACP. Of these, n = 27 had data eligible for analysis (n = 8 in round 1 and n = 19 in rounds 2 and 3). Interviews were audio-recorded and analyzed using the constant comparison method. Three reviewers independently listened to the interviews, summarized findings and discussed discrepancies until consensus was achieved.ResultsItem reduction from key content expert review and conversation analysis helped decrease number of items from 116 in the original ACP Engagement Survey to 24–38 in the new BACPACS depending on branching of responses. For the think aloud study, three rounds of interviews were needed until saturation for patient clarity was achieved. The understanding of ACP as a construct, survey response options, instructions and terminology pertaining to patient engagement in ACP warranted further clarification.ConclusionsConversation analysis, content expert review and think aloud cognitive interviewing were useful in refining the new survey instrument entitled BACPACS. We found evidence for both content and response process validity for this new tool.

Assessment of motivational interviewing: a qualitative study of response process validity, content validity and feasibility of the motivational interviewing target scheme (MITS) in general practice

BackgroundThe Motivational Interviewing target Scheme (MITS) is an instrument to assess competency in Motivational Interviewing (MI) and can be used to assess MI in long and brief consultations. In this qualitative study we examined two sources of the Unified Model of Validity, the current standard of assessment validation, in the context of General Practice. We collected evidence concerning response process validity and content validity of the MITS in general practice. Furthermore, we investigated its feasibility.MethodsAssessing consultations of General Practitioners and GP-trainees (GPs), the assessors systematically noted down their considerations concerning the scoring process and the content of the MITS in a semi-structured questionnaire. Sampling of the GPs was based on maximum variation and data saturation was used as a stopping criterion. An inductive approach was used to analyse the data. In response to scoring problems the score options were adjusted and all consultations were assessed using the original and the adjusted score options.ResultsTwenty seven assessments were needed to reach data saturation. In most consultations, the health behaviour was not the reason for encounter but was discussed on top of discussing physical problems. The topic that was most discussed in the consultations was smoking cigarettes. The adjusted score options increased the response process validity; they were more in agreement with theoretical constructs and the observed quality of MI in the consultations. Concerning content validity, we found that the MITS represents the broad spectrum and the current understanding of MI. Furthermore, the MITS proved to be feasible to assess MI in brief consultations in general practice.ConclusionsBased on the collected evidence the MITS seems a promising instrument to measure MI interviewing in brief consultations. The evidence gathered in this study lays the foundation for research into other aspects of validation.

A Simulation-based Approach to Measuring Team Situational Awareness in Emergency Medicine: A Multicenter, Observational Study.

Team situational awareness (TSA) is critical for effective teamwork and supports dynamic decision making in unpredictable, time-pressured situations. Simulation provides a platform for developing and assessing TSA, but these efforts are limited by suboptimal measurement approaches. The objective of this study was to develop and evaluate a novel approach to TSA measurement in interprofessional emergency medicine (EM) teams. We performed a multicenter, prospective, simulation-based observational study to evaluate an approach to TSA measurement. Interprofessional emergency medical teams, consisting of EM resident physicians, nurses, and medical students, were recruited from the University of Washington (Seattle, WA) and Wayne State University (Detroit, MI). Each team completed a simulated emergency resuscitation scenario. Immediately following the simulation, team members completed a TSA measure, a team perception of shared understanding measure, and a team leader effectiveness measure. Subject matter expert reviews and pilot testing of the TSA measure provided evidence of content and response process validity. Simulations were recorded and independently coded for team performance using a previously validated measure. The relationships between the TSA measure and other variables (team clinical performance, team perception of shared understanding, team leader effectiveness, and team experience) were explored. The TSA agreement metric was indexed by averaging the pairwise agreement for each dyad on a team and then averaging across dyads to yield agreement at the team level. For the team perception of shared understanding and team leadership effectiveness measures, individual team member scores were aggregated within a team to create a single team score. We computed descriptive statistics for all outcomes. We calculated Pearson's product-moment correlations to determine bivariate correlations between outcome variables with two-tailed significance testing (p < 0.05). A total of 123 participants were recruited and formed three-person teams (n = 41 teams). All teams completed the assessment scenario and postsimulation measures. TSA agreement ranged from 0.19 to 0.9 and had a mean (±SD) of 0.61 (±0.17). TSA correlated with team clinical performance (p < 0.05) but did not correlate with team perception of shared understanding, team leader effectiveness, or team experience. Team situational awareness supports adaptive teams and is critical for high reliability organizations such as healthcare systems. Simulation can provide a platform for research aimed at understanding and measuring TSA. This study provides a feasible method for simulation-based assessment of TSA in interdisciplinary teams that addresses prior measure limitations and is appropriate for use in highly dynamic, uncertain situations commonly encountered in emergency department systems. Future research is needed to understand the development of and interactions between individual-, team-, and system (distributed)-level cognitive processes.

Assessing Quality of Kindergarten Classrooms in Singapore: Psychometric Properties of the Early Childhood Environment Rating Scale—Revised

The early childhood sector in Singapore has witnessed vast changes in the past two decades. One of the key policy aims is to improve classroom quality. To ensure a rigorous evaluation of the quality of early childhood environments in Singapore, it is important to determine whether commonly used assessments of quality are valid indicators across different national and community contexts. This study investigated the validity of the Early Childhood Environment Rating Scale—Revised (ECERS-R) in Singapore focused on three major aspects: response process, structural, and criterion validity. Data were collected from 108 classrooms of the main providers of kindergarten programs in Singapore. Initial analyses showed that the quality indicators in most of the ECERS-R items were disordered, which indicated problems with the hypothesized response process underlying the standard ECERS-R scoring system in which high-level indicators are only rated if low-level indicators are met (stop-scoring). To deal with the problem of the “stop-scoring” method, we used the data from a full-scoring method (in which every indicator on every item was rated) in subsequent factor analyses to investigate the internal structure of the ECERS-R. Twenty-two of the 34 items were retained and found to load on two factors, one relating to activities/materials and the other related to language/interactions. Both of the identified subscales and the combined scale showed good internal consistency. The factor score on the combined scale mirrored the ECERS-R full score and could discriminate between classrooms in the top and bottom quartiles of quality scores. Findings on the relationship between the identified factors to the teacher–child ratio and to an alternative measure of classroom quality provided further evidence for criterion validity of the ECERS-R. Implications for the future development and use of the ECERS-R are discussed.

Development and Assessment of a Novel Task Trainer and Targeting Tasks for Ultrasound-guided Invasive Procedures

The American Institute of Ultrasound in Medicine (AIUM) recommends that all providers performing ultrasound-guided invasive procedures be competent in a core set of guidance skills common to all ultrasound-guided procedures, including in-plane and out-of-plane needle guidance and needle imaging optimization techniques such as probe translation, rotation, and heel-toe standoff. To allow for the practice and assessment of these core skills, we have created a novel task trainer and set of targeting tasks, and sought to obtain validity evidence in the content and response process domains for this training and assessment system according to the Standards for Educational and Psychological Testing. We have constructed an ultrasound-guided invasive procedure training system and five targeting tasks that focused on the needle guidance skills outlined by the AIUM. All tasks were performed by obstetrics and gynecology or maternal-fetal medicine physicians with and without experience in ultrasound-guided invasive procedures during a series of simulation workshops. All participants completed a survey regarding the trainer's and the tasks' usefulness in the training of inexperienced physicians. The physicians who completed the tasks had favorable views of task trainer and curriculum. The targeting curriculum was felt to allow practice of all of the core guidance skills outlined by the AIUM. The average response provided for all of the tasks was 4.0 or greater, with half of the items having an average response of 4.5 or higher. We have constructed a task trainer that incorporates all of the core skills outlined by the AIUM. All five tasks received very favorable reviews from both experienced and inexperienced providers. Taken together, our findings suggest they have strong content and response process validity evidence.

Can Residents Assess Other Providers' Infant Lumbar Puncture Skills?: Validity Evidence for a Global Rating Scale and Subcomponent Skills Checklist.

The aims of this study were to provide validity evidence for infant lumbar puncture (ILP) checklist and global rating scale (GRS) instruments when used by residents to assess simulated ILP performances and to compare these metrics to previously obtained attending rater data. In 2009, the International Network for Simulation-based Pediatric Innovation, Research, and Education (INSPIRE) developed checklist and GRS scoring instruments, which were previously validated among attending raters when used to assess simulated ILP performances. Video recordings of 60 subjects performing an LP on an infant simulator were collected; 20 performed by subjects in 3 categories (beginner, intermediate, and expert). Six blinded pediatric residents independently scored each performance (3 via the GRS, 3 via the checklist). Four of the 5 domains of validity evidence were collected: content, response process, internal structure (reliability and discriminant validity), and relations to other variables. Evidence for content and response process validity is presented. When used by residents, the checklist performed similarly to what was found for attending raters demonstrating good internal consistency (Cronbach α = 0.77) and moderate interrater agreement (intraclass correlation coefficient = 0.47). Residents successfully discerned beginners (P < 0.01, effect size = 2.1) but failed to discriminate between expert and intermediate subjects (P = 0.68, effect size = 0.34). Residents, however, gave significantly higher GRS scores than attending raters across all subject groups (P < 0.001). Moderate correlation was found between GRS and total checklist scores (P = 0.49, P < 0.01). This study provides validity evidence for the checklist instrument when used by pediatric residents to assess ILP performances. Compared with attending raters, residents appeared to over-score subjects on the GRS instrument.

Improving Adhesion of Internal Jugular Dressings in the Intensive Care Unit

To date, there is no published, psychometrically validated, short peripheral intravenous catheter (PIVC) insertion skills checklist. Creating a valid, reliable, and generalizable checklist to measure PIVC skill is a key step in assessing baseline competence and skill mastery. Based on recognized standards and best practices, the PIVC Insertion Skills Checklist was developed to measure all the steps necessary for a best practice PIVC insertion. This includes the entire process from reading the prescriber's orders to documentation and, if the first attempt is unsuccessful, a second attempt option. Content validity was established using 3 infusion therapy experts. Evidence in support of response process validity is described. The PIVC Insertion Skills Checklist was used by 8 trained raters to assess the PIVC insertion skills, in a simulated environment, of 63 practicing clinicians working on medical and surgical units in a US teaching hospital. Internal consistency of the PIVC Insertion Skills Checklist was α = 0.84. Individual item intraclass correlation coefficients (ICCs) between rater and gold standard observations ranged from −0.01 to 1.00 and total score ICC was 0.99 (95% confidence interval, 0.99-0.99). The current study offers validity and reliability evidence to support the use of the PIVC Insertion Skills Checklist to measure PIVC insertion skill of clinicians in a simulated environment.

The Expansion of a Nurse-led Peripherally Inserted Central Catheter (PICC) Team to a Vascular Resource Team to Insert Peripheral Intravenous Catheters

To date, there is no published, psychometrically validated, short peripheral intravenous catheter (PIVC) insertion skills checklist. Creating a valid, reliable, and generalizable checklist to measure PIVC skill is a key step in assessing baseline competence and skill mastery. Based on recognized standards and best practices, the PIVC Insertion Skills Checklist was developed to measure all the steps necessary for a best practice PIVC insertion. This includes the entire process from reading the prescriber's orders to documentation and, if the first attempt is unsuccessful, a second attempt option. Content validity was established using 3 infusion therapy experts. Evidence in support of response process validity is described. The PIVC Insertion Skills Checklist was used by 8 trained raters to assess the PIVC insertion skills, in a simulated environment, of 63 practicing clinicians working on medical and surgical units in a US teaching hospital. Internal consistency of the PIVC Insertion Skills Checklist was α = 0.84. Individual item intraclass correlation coefficients (ICCs) between rater and gold standard observations ranged from −0.01 to 1.00 and total score ICC was 0.99 (95% confidence interval, 0.99-0.99). The current study offers validity and reliability evidence to support the use of the PIVC Insertion Skills Checklist to measure PIVC insertion skill of clinicians in a simulated environment.

Development and Testing of a Short Peripheral Intravenous Catheter Insertion Skills Checklist

To date, there is no published, psychometrically validated, short peripheral intravenous catheter (PIVC) insertion skills checklist. Creating a valid, reliable, and generalizable checklist to measure PIVC skill is a key step in assessing baseline competence and skill mastery. Based on recognized standards and best practices, the PIVC Insertion Skills Checklist was developed to measure all the steps necessary for a best practice PIVC insertion. This includes the entire process from reading the prescriber's orders to documentation and, if the first attempt is unsuccessful, a second attempt option. Content validity was established using 3 infusion therapy experts. Evidence in support of response process validity is described. The PIVC Insertion Skills Checklist was used by 8 trained raters to assess the PIVC insertion skills, in a simulated environment, of 63 practicing clinicians working on medical and surgical units in a US teaching hospital. Internal consistency of the PIVC Insertion Skills Checklist was α = 0.84. Individual item intraclass correlation coefficients (ICCs) between rater and gold standard observations ranged from −0.01 to 1.00 and total score ICC was 0.99 (95% confidence interval, 0.99-0.99). The current study offers validity and reliability evidence to support the use of the PIVC Insertion Skills Checklist to measure PIVC insertion skill of clinicians in a simulated environment.

Deployment of a Time of Catheters: Experience Report

To date, there is no published, psychometrically validated, short peripheral intravenous catheter (PIVC) insertion skills checklist. Creating a valid, reliable, and generalizable checklist to measure PIVC skill is a key step in assessing baseline competence and skill mastery. Based on recognized standards and best practices, the PIVC Insertion Skills Checklist was developed to measure all the steps necessary for a best practice PIVC insertion. This includes the entire process from reading the prescriber's orders to documentation and, if the first attempt is unsuccessful, a second attempt option. Content validity was established using 3 infusion therapy experts. Evidence in support of response process validity is described. The PIVC Insertion Skills Checklist was used by 8 trained raters to assess the PIVC insertion skills, in a simulated environment, of 63 practicing clinicians working on medical and surgical units in a US teaching hospital. Internal consistency of the PIVC Insertion Skills Checklist was α = 0.84. Individual item intraclass correlation coefficients (ICCs) between rater and gold standard observations ranged from −0.01 to 1.00 and total score ICC was 0.99 (95% confidence interval, 0.99-0.99). The current study offers validity and reliability evidence to support the use of the PIVC Insertion Skills Checklist to measure PIVC insertion skill of clinicians in a simulated environment.

Peripheral Vascular Catheter Research: A Scoping Review

To date, there is no published, psychometrically validated, short peripheral intravenous catheter (PIVC) insertion skills checklist. Creating a valid, reliable, and generalizable checklist to measure PIVC skill is a key step in assessing baseline competence and skill mastery. Based on recognized standards and best practices, the PIVC Insertion Skills Checklist was developed to measure all the steps necessary for a best practice PIVC insertion. This includes the entire process from reading the prescriber's orders to documentation and, if the first attempt is unsuccessful, a second attempt option. Content validity was established using 3 infusion therapy experts. Evidence in support of response process validity is described. The PIVC Insertion Skills Checklist was used by 8 trained raters to assess the PIVC insertion skills, in a simulated environment, of 63 practicing clinicians working on medical and surgical units in a US teaching hospital. Internal consistency of the PIVC Insertion Skills Checklist was α = 0.84. Individual item intraclass correlation coefficients (ICCs) between rater and gold standard observations ranged from −0.01 to 1.00 and total score ICC was 0.99 (95% confidence interval, 0.99-0.99). The current study offers validity and reliability evidence to support the use of the PIVC Insertion Skills Checklist to measure PIVC insertion skill of clinicians in a simulated environment.

Using High Flow Flushing Technique (HFFT) as Initial Intervention for Malpositioned Peripherally Inserted Central Catheters (PICC's) in the Adult Population

To date, there is no published, psychometrically validated, short peripheral intravenous catheter (PIVC) insertion skills checklist. Creating a valid, reliable, and generalizable checklist to measure PIVC skill is a key step in assessing baseline competence and skill mastery. Based on recognized standards and best practices, the PIVC Insertion Skills Checklist was developed to measure all the steps necessary for a best practice PIVC insertion. This includes the entire process from reading the prescriber's orders to documentation and, if the first attempt is unsuccessful, a second attempt option. Content validity was established using 3 infusion therapy experts. Evidence in support of response process validity is described. The PIVC Insertion Skills Checklist was used by 8 trained raters to assess the PIVC insertion skills, in a simulated environment, of 63 practicing clinicians working on medical and surgical units in a US teaching hospital. Internal consistency of the PIVC Insertion Skills Checklist was α = 0.84. Individual item intraclass correlation coefficients (ICCs) between rater and gold standard observations ranged from −0.01 to 1.00 and total score ICC was 0.99 (95% confidence interval, 0.99-0.99). The current study offers validity and reliability evidence to support the use of the PIVC Insertion Skills Checklist to measure PIVC insertion skill of clinicians in a simulated environment.