Temporal Stability Reliability Research Articles

Assessment of kinematic concepts comprehension: A systematic review

<span lang="EN-US">A systematic study on the development of the assessment instrument of the concept of kinematics which is one of the basic concepts of physics has not yet been conducted. This systematic study reported the characteristics, quality, effectiveness, and efficiency of the instrument assessment regarding the concept of kinematics. This systematic review method has used a systematic approach based on the practical guidelines of systematic reviews by Petticrew and Roberts and has used PRISMA diagrams for the process of searching and selecting articles, who reviewed 12 articles published between 1985 and 2021. The research findings show that the instrument of concept mastery of kinematics using multiple-choice formats, while others combine the true-false and multiple-choice formats. Almost all instruments of mastery of the concept of widely used kinematics have reported validity assessment (content validity, construct validity) and internal consistency reliability. However, predictive validity, discriminatory validity, and temporal stability reliability have not been reported. Other results indicate that the development of further assessments must pay attention to five important aspects: the validity and reliability of instruments, the scopes and specificity of the concept of kinematics, the conformity with the intellectual level of students, the effectiveness and efficiency of instrument assessment, and the fulfillment of research gaps. This study can help educators and researchers choose and develop an appropriate instrument assessment to reveal students’ understanding of the concept of kinematics.</span>

Peyronie's Disease Questionnaire (PDQ): Spanish translation and validation.

The Peyronie's Disease Questionnaire (PDQ) is a 15-question self-reported questionnaire that evaluates the severity and physical and psychosexual issues of Peyronie's disease (PD) symptoms in three scales: "psychological and physical symptoms," "penile pain," and "symptom bother." Previous studies validated the PDQ US version and confirmed its test-retest reliability and responsiveness. The aim is to translate and validate the Spanish version of the PDQ to be used in the clinical practice and in PD research studies in Spain. A non-interventional, observational study with 160 PD patients was conducted. Patients included from four healthcare centers in Spain and completed the PDQ in two study visits separated for a period of 4-7 days from March 2018 to June 2019. Patients received no type of treatment or intervention. Different statistical tests were applied to the data in order to validate the structural and construct of the PDQ, as well as its internal reliability, temporal stability reliability, reliability between observers, and test-retest reliability. Cronbach's alpha over 0.9 showed good internal consistency. We found an ICC agreement of 0.82 (test-retest) for the three scales of the Spanish version of the PDQ, which demonstrates good reliability. When comparing Visit 1 and Visit 2 questionnaires mean scores, the PDQ showed non-significant differences, as expected because no intervention or treatment was administered to the patients between visits. Translation and validation of the PDQ for the Spanish population makes available a valid, useful, and reliable tool to properly evaluate quality of life of men suffering PD.

A Comprehensive Review of Instruments Measuring Attitudes Toward Science

The development of attitudes toward science instruments has recently emerged in science education research. However, a comprehensive review of their psychometric properties, using currently accepted assessment standards, has not yet been completed. Consequently, this review discusses the validity and reliability of 18 measures published between 2005 and 2019 in leading science education journals. Findings showed that construct validity and internal consistency reliability was reported for all instruments; however, evidence for predictive validity and temporal stability reliability was rather scarce, which could limit their use in intervention and correlational type of studies. Similarly, content validity was found to be underreported. Consequently, the relevance, comprehensiveness, and comprehensibility of the items in some instruments are currently unknown and yet to be established in future studies. Finally, there is a gap in the literature regarding instruments that can be used across different countries and scientific disciplines, which could restrict accumulative and comparative results worldwide. Since the use of valid and reliable measurement instruments is a crucial aspect of educational research, the findings of this study could be useful in assisting researchers and practitioners in selecting the most appropriate measure for different research designs.

Development and preliminary validation of a post-fistula repair reintegration instrument among Ugandan women

BackgroundObstetric fistula is a debilitating and traumatic birth injury affecting 2–3 million women globally, mostly in sub-Saharan Africa and Asia. Affected women suffer physically, psychologically and socioeconomically. International efforts have increased access to surgical treatment, yet attention to a holistic outcome of post-surgical rehabilitation is nascent. We sought to develop and pilot test a measurement instrument to assess post-surgical family and community reintegration.MethodsWe conducted an exploratory sequential mixed-methods study, beginning with 16 in-depth interviews and four focus group discussions with 17 women who underwent fistula surgery within two previous years to inform measure development. The draft instrument was validated in a longitudinal cohort of 60 women recovering from fistula surgery. Qualitative data were analyzed through thematic analysis. Socio-demographic characteristics were described using one-way frequency tables. We used exploratory factor analysis to determine the latent structure of the scale, then tested the fit of a single higher-order latent factor. We evaluated internal consistency and temporal stability reliability through Raykov’s ρ and Pearson’s correlation coefficient, respectively. We estimated a series of linear regression models to explore associations between the standardized reintegration measure and validated scales representing theoretically related constructs.ResultsThemes central to women’s experiences following surgery included resuming mobility, increasing social interaction, improved self-esteem, reduction of internalized stigma, resuming work, meeting their own needs and the needs of dependents, meeting other expected and desired roles, and negotiating larger life issues. We expanded the Return to Normal Living Index to reflect these themes. Exploratory factor analysis suggested a four-factor structure, titled ‘Mobility and social engagement’, ‘Meeting family needs’, ‘Comfort with relationships’, and ‘General life satisfaction’, and goodness of fit statistics supported a higher-order latent variable of ‘Reintegration.’ Reintegration score correlated significantly with quality of life, depression, self-esteem, stigma, and social support in theoretically expected directions.ConclusionAs more women undergo surgical treatment for obstetric fistula, attention to the post-repair period is imperative. This preliminary validation of a reintegration instrument represents a first step toward improving measurement of post-surgical reintegration and has important implications for the evidence base of post-surgical reintegration epidemiology and the development and evaluation of fistula programming.

Validation of a Teaching Effectiveness Assessment in Psychiatry Continuing Medical Education.

Little is known about factors associated with effective continuing medical education (CME) in psychiatry. The authors aimed to validate a method to assess psychiatry CME teaching effectiveness and to determine associations between teaching effectiveness scores and characteristics of presentations, presenters, and participants. This cross-sectional study was conducted at the Mayo Clinic Psychiatry Clinical Reviews and Psychiatry in Medical Settings. Presentations were evaluated using an eight-item CME teaching effectiveness instrument, its content based on previously published instruments. Factor analysis, internal consistency and interrater reliabilities, and temporal stability reliability were calculated. Associations were determined between teaching effectiveness scores and characteristics of presentations, presenters, and participants. In total, 364 participants returned 246 completed surveys (response rate, 67.6%). Factor analysis revealed a unidimensional model of psychiatry CME teaching effectiveness. Cronbach α for the instrument was excellent at 0.94. Item mean score (SD) ranged from 4.33 (0.92) to 4.71 (0.59) on a 5-point scale. Overall interrater reliability was 0.84 (95% CI, 0.75-0.91), and temporal stability was 0.89 (95% CI, 0.77-0.97). No associations were found between teaching effectiveness scores and characteristics of presentations, presenters, and participants. This study provides a new, validated measure of CME teaching effectiveness that could be used to improve psychiatry CME. In contrast to prior research in other medical specialties, CME teaching effectiveness scores were not associated with use of case-based or interactive presentations. This outcome suggests the need for distinctive considerations regarding psychiatry CME; a singular approach to CME teaching may not apply to all medical specialties.

A review and analysis of research on the test-retest reliability of professional judgment

This paper analyzes existing research on the test–retest reliability of human judgment, i.e. the extent to which a judge makes identical judgments when presented with identical stimuli on two occasions. Only research involving professional judges who make experimental judgments in a reasonable analog of their everyday experience is included. Studies of both internal consistency reliability and temporal stability reliability are analyzed (where the former refers to the inclusion of repeat stimuli in the same experimental session, and the latter refers to the repeating of the experimental task from a few days to several months later). It is found that (1) the test–retest reliability literature is concentrated in four substantive judgment areas (medicine/psychology, meteorology, human resources management, and business), (2) the literature is extremely variable in terms of research approach/design, the determinants or correlates of test–retest reliability that have been studied, and the quality of the execution and analysis, and (3) mean test–retest reliability differs across both substantive judgment areas and the internal consistency versus temporal stability distinction. An inescapable conclusion from the analysis is that our knowledge of this fundamental property of human judgment is quite meager. Therefore, the paper concludes with suggestions about future research that would address test–retest reliability more systematically. Copyright © 2000 John Wiley & Sons, Ltd.