Single-measure Intraclass Correlation Research Articles

Test-retest reliability of selected items of Health Behaviour in School-aged Children (HBSC) survey questionnaire in Beijing, China

BackgroundChildren's health and health behaviour are essential for their development and it is important to obtain abundant and accurate information to understand young people's health and health behaviour. The Health Behaviour in School-aged Children (HBSC) study is among the first large-scale international surveys on adolescent health through self-report questionnaires. So far, more than 40 countries in Europe and North America have been involved in the HBSC study. The purpose of this study is to assess the test-retest reliability of selected items in the Chinese version of the HBSC survey questionnaire in a sample of adolescents in Beijing, China.MethodsA sample of 95 male and female students aged 11 or 15 years old participated in a test and retest with a three weeks interval. Student Identity numbers of respondents were utilized to permit matching of test-retest questionnaires. 23 items concerning physical activity, sedentary behaviour, sleep and substance use were evaluated by using the percentage of response shifts and the single measure Intraclass Correlation Coefficients (ICC) with 95% confidence interval (CI) for all respondents and stratified by gender and age. Items on substance use were only evaluated for school children aged 15 years old.ResultsThe percentage of no response shift between test and retest varied from 32% for the item on computer use at weekends to 92% for the three items on smoking. Of all the 23 items evaluated, 6 items (26%) showed a moderate reliability, 12 items (52%) displayed a substantial reliability and 4 items (17%) indicated almost perfect reliability. No gender and age group difference of the test-retest reliability was found except for a few items on sedentary behaviour.ConclusionsThe overall findings of this study suggest that most selected indicators in the HBSC survey questionnaire have satisfactory test-retest reliability for the students in Beijing. Further test-retest studies in a large and diverse sample, as well as validity studies, should be considered for the future Chinese HBSC study.

Creation and Initial Evaluation of a Stool Form Scale for Children

To develop a pediatric stool form rating scale and determine its interrater reliability, intrarater reliability, and agreement among pediatric gastroenterologists. An ordinal stool scale with 5 categorical stool form types was created on the basis of the Bristol Stool Form Scale, and 32 color 2-dimensional stool photographs were shown to 14 pediatric gastroenterologists. Each gastroenterologist rated the stool form depicted in each photograph with the modified stool scale. Ten gastroenterologists agreed to rerate the stool form depicted in each photograph a minimum of 6 months after the first rating. A total of 448 ratings were completed; 430 (94%) of all ratings were within at least 1 category type of the most common (modal) rating for each photograph. Eight (25%) stool photographs had complete agreement among all raters. Interrater and intrarater reliability was high with a single measure intraclass correlation of 0.85 (95% confidence interval: 0.78-0.91; P<.001) and 0.87 (95% confidence interval: 0.81-0.92; P<.001), respectively. A modified pediatric Bristol Stool Form Scale provided a high degree of interrater reliability, intrarater reliability, and agreement among pediatric gastroenterologists.

A Simplified Method of T1ρ Mapping in Clinical Assessment of Knee Joint

we assessed the clinical utility of our proposed simplified method for T(1)ρ mapping calculations. ten healthy subjects underwent scanning on a 3-tesla magnetic resonance system using an 8-channel phased-array coil. For each subject, we obtained sagittal T(1)ρ-prepared images using 5 different time of spin-lock pulses (TSL=1, 20, 40, 60, and 80 ms), produced conventional T(1)ρ maps (cT(1)ρ maps) using all TSLs, and recomputed our proposed simplified T(1)ρ maps (sT(1)ρ maps) using a decreasing number of TSLs (from 4 to 2). We then investigated the differences and correlations in T(1)ρ values of the tissues obtained using different numbers of spin-lock times. there was a strong positive correlation (single measure intraclass correlation coefficient=0.948; 95% confidence interval=0.911 to 0.970) in T(1)ρ values of tissues between the cT(1)ρ and sT(1)ρ [1, 80] maps. The 2 maps were comparable, though there was a small difference in T(1)ρ value between the two. The total scan time to acquire the data from 5 spin-lock times was 16 min 15 s. Similarity of the T(1)ρ [1, 80] map with the conventional approach reduced scan time by 60%, to 6 min 30 s. the clinical relevance of our proposed simplified method is potentially similar to that of the conventional method, and our method requires a shorter examination time and generally preserves the reliability of the T(1)ρ relaxation time of the tissues.

A MnCl 2-based phantom for IVD hydration status determination using MRI: Pre-clinical reliability analyses

Introduction Non-biological, yet signal-responsive, MR phantom materials are becoming increasingly commonplace. One such novel agent, semi-solid manganese chloride (MnCl 2), has recently been described as a potential calibration standard for direct assessment of in vivo cartilage fluid content. Given the established correlation between intervertebral disc (IVD) hydration state and physiologic spinal functioning, such a tool, allowing calibration for ‘quantitative’ appraisal of disc fluid content, has many potential applications. The purpose of this study was to demonstrate MR signal-to-noise ratio (SNR) measurement reliability of a novel MnCl 2-based signal calibration phantom for in vivo disc hydration analysis, using a 10 × 10 inter- and intra-observer reliability analysis in the pre-clinical setting. Materials and methods A series of novel MnCl 2 calibration phantoms were imaged to assess intra-/inter-observer reliability during measurement of signal intensity. The phantoms were imaged under ten different MR sequences, generating 75 signal regions from which SNR values were measured. Inter-observer reliability was tested by inviting ten individuals to obtain signal measurements from each image, on a single occasion. To test intra-observer reliability, a single participant was asked to record measurements of the same features on ten separate occasions. Results 1425 Discrete measurement points were available for combined reliability analyses. Single-measure intraclass correlation coefficients showed high measurement agreement, with both intra- and inter-observer values approaching 1.00. Conclusion This study demonstrates that signal measurements can be obtained using MnCl 2 disc phantoms, with a high degree of observer reliability, supporting their use as a signal calibration standard during orthopaedic MR-based cartilage imaging.

Q-sort assessment vs visual analog scale in the evaluation of smile esthetics

This study was designed to compare the reliability of the Q-sort and visual analog scale (VAS) methods for the assessment of smile esthetics. Furthermore, agreement between orthodontists and parents of orthodontic patients, and between male and female raters, was assessed in terms of subjective evaluation of the smile. Clinical photographs and digital video captures of 48 orthodontically treated patients were rated by 2 panels: 25 experienced orthodontists (15 men, 10 women) and 20 parents of the patients (8 men, 12 women). Interrater reliability of the Q-sort and VAS methods was evaluated by using single-measure and average-measure intraclass correlation (ICC). Kappa agreement and the McNemar test were used to evaluate agreement between orthodontists and parents, and between men and women, for "attractive" and "unattractive" images of smiles captured with clinical photography. The single-measure ICC coefficients showed fair to good reliability of the Q-sort and poor reliability of the VAS for measuring esthetic preferences of an individual orthodontist or parent. Both rating groups agreed significantly (P >0.05) on the total percentage of "attractive" images of smiles captured with clinical photography. Men and women, however, significantly disagreed on the total percentages of "attractive" and "unattractive" smiles. Women rated higher percentages of both image groups as "attractive" than did their male counterparts. The Q-sort was more reliable than the VAS for measuring smile esthetics. Orthodontists and parents of orthodontic patients agreed with respect to "attractive" and "unattractive" smiles. Men and women agreed poorly with respect to "attractive" and "unattractive" smiles.

Intertester Reliability and Diagnostic Validity of the Cervical Flexion-Rotation Test

Objective This article evaluates reliability and diagnostic validity of the cervical flexion-rotation test (FRT) to discriminate subjects with headache because of C1/2 dysfunction. In addition, this study evaluates agreement between experienced and inexperienced examiners. Methods These were 2 single blind comparative measurement study designs. In study 1, 2 experienced blinded examiners evaluated the FRT in 10 asymptomatic controls, 20 subjects with cervicogenic headache (CeH) where C1/2 was the primary dysfunctional level, and 10 subjects with CeH but without C1/2 as the primary dysfunctional level. In study 2, 2 inexperienced and 1 experienced blinded examiners evaluated the FRT in 12 subjects with CeH and 12 asymptomatic controls. Examiners were required to state whether the FRT was positive and also to determine range of rotation using a goniometer. An analysis of variance with planned orthogonal comparison, single measure intraclass correlation coefficient (2,1), and Bland-Altman plot were used to analyze FRT range of rotation between the examiners. Sensitivity, specificity, and examiner agreement for test interpretation were analyzed using cross tabulation and κ. Results In study 1, sensitivity and specificity of the FRT was 90% and 88% with 92% agreement for experienced examiners ( P < .001). Overall diagnostic accuracy was 89% ( P < .001) and κ = 0.85. In study 2, for inexperienced examiners, FRT mobility was significantly greater than for experienced examiners, but sensitivity, specificity, agreement, and κ values were all within clinically acceptable levels. Conclusions The FRT can be used accurately and reliably by inexperienced examiners and may be a useful aid in CeH evaluation.

REPRODUCIBILITY OF POWER PRODUCTION DURING SPRINT ARM ERGOMETRY

Sprint tests are frequently used to evaluate between- subject differences and can provide a valuable insight into performance capacity. The present study determined the reproducibility of peak and mean power output during upper-body sprints. After familiarization 25 men (mean [± SD] age 29 [6] years, body mass 82.8 [12.7] kg and height 1.76 [0.05] m) completed 2 20-second upper-body sprint tests using an adapted cycle ergometer. Mean (± SD) values of all power (uncorrected and corrected) measurements achieved during the 2 tests were checked for systematic bias using separate paired t-tests. Test- retest reproducibility was examined using coefficients of variation and single-measure intraclass correlation coefficients, as well as an assessment of the typical (random) error and the 95% limits of agreement. The value of corrected peak power (628 [167] W) was higher (p < 0.05) compared with the uncorrected value (509 [109] W). Values of corrected (465 [95] W) and uncorrected (444 [87] W) mean power were similar (p > 0.05). The mean bias value for all power parameters equated to less than =1% of the absolute values of power measured. Intraclass correlation coefficients for all data sets ranged from 0.97 to 0.98. Coefficients of variation for uncorrected and corrected values of peak power were 2.8 and 4.5%, while corresponding values for mean power were 2.9 and 3.2%, respectively. The reproducibility of all power indices was below 5%. The results of this study indicate that both uncorrected and corrected measurements of peak power output and mean power output can be used to assess performance during sprint arm ergometry.

The Reliability of a Posterior-to-Anterior Spinal Stiffness Measuring System in a Population of Patients With Low Back Pain

Objective A system for measuring posterior-to-anterior spinal stiffness (PAS) was developed for use in clinical trials of manipulation for low back pain. The reliability of this device is under investigation in this study. Methods The PAS system uses electronic sensors to record displacement and force while a human operator provides the force of indentation. A test-retest design was used with measures repeated by the same operator within 5 minutes. Posterior-to-anterior loads were applied to each lumbar spinous process of patients lying prone on a hard flat bench. Force and displacement were recorded and used to calculate PAS. Results The subjects consisted of 22 males and 14 females; average age was 49.1 years (SD, 14.2). All subjects had low back pain of at least 4 weeks duration, with mean Roland-Morris scores of 7.6 (SD, 3.3). Spinal stiffness ranged from 4 to 26 N/mm (average, 11.2; SD, 3.5). Stiffness in the first and second tests varied on the average by 0.31 N/mm ( P = .03). Standard error of the measurement was 1.62 N/mm. The single measures intraclass correlation coefficient (3,1) was 0.790 (95% confidence interval, 0.739-0.832). Conclusions The equipment and method produced repeatable results over the short-term. The system may be sensitive enough to detect changes in spinal stiffness that occur with care.

Psychometric properties of the Multifactor Leadership Questionnaire among nurses

This paper reports a study investigating the psychometric properties of the Multifactor Leadership Questionnaire among nurses. Although the Multifactor Leadership Questionnaire is one of the most widely used instruments to measure the multifactor leadership theory in organizational sciences, inconsistent research findings have been reported for its psychometric properties. Little is known about these properties. Data were gathered by postal questionnaires in 2001-2002 with nurses working in different healthcare organizations in Finland. A follow-up study was performed 1-year later. The sample consisted of 601 nurses and nurse leaders, and the follow-up study had 78 respondents. The internal consistency of the Multifactor Leadership Questionnaire was explored using Cronbach's alpha coefficient and item-analysis, and the test-retest reliability using Pearson product moment correlation coefficient and intra-class correlation coefficient (single measure intra-class correlation coefficient). The factor structure was studied using exploratory and confirmatory factor analyses. The internal consistencies of the leadership subscales were satisfactory. The Multifactor Leadership Questionnaire was fairly stable measured at the 1-year interval. Although the data fail to support the full nine-factor model, a reduced set of items from the Multifactor Leadership Questionnaire appears to show evidence of the three- and six-factor structures. Exploratory factor analysis showed evidence for the three-factor structure consisting of three distinctive, yet partly related leadership subscales. Confirmatory factor analysis showed evidence for the six-factor structure. The psychometric data suggest that a modified version of the Multifactor Leadership Questionnaire is a highly suitable instrument to measure multidimensional nursing leadership. Validity and reliability were supported. The results can be used in studies applying the Multifactor Leadership Questionnaire.

Reproducibility of Power Production During Sprint Arm Ergometry

Sprint tests are frequently used to evaluate between-subject differences and can provide a valuable insight into performance capacity. The present study determined the reproducibility of peak and mean power output during upper-body sprints. After familiarization 25 men (mean [+/- SD] age 29 [6] years, body mass 82.8 [12.7] kg and height 1.76 [0.05] m) completed 2 20-second upper-body sprint tests using an adapted cycle ergometer. Mean (+/- SD) values of all power (uncorrected and corrected) measurements achieved during the 2 tests were checked for systematic bias using separate paired t-tests. Test-retest reproducibility was examined using coefficients of variation and single-measure intraclass correlation coefficients, as well as an assessment of the typical (random) error and the 95% limits of agreement. The value of corrected peak power (628 [167] W) was higher (p < 0.05) compared with the uncorrected value (509 [109] W). Values of corrected (465 [95] W) and uncorrected (444 [87] W) mean power were similar (p > 0.05). The mean bias value for all power parameters equated to less than +/-1% of the absolute values of power measured. Intraclass correlation coefficients for all data sets ranged from 0.97 to 0.98. Coefficients of variation for uncorrected and corrected values of peak power were 2.8 and 4.5%, while corresponding values for mean power were 2.9 and 3.2%, respectively. The reproducibility of all power indices was below 5%. The results of this study indicate that both uncorrected and corrected measurements of peak power output and mean power output can be used to assess performance during sprint arm ergometry.

Event‐related functional near‐infrared spectroscopy (fNIRS) based on craniocerebral correlations: Reproducibility of activation?

The purpose of the present study was to assess the retest reliability of cortical activation detected by event-related functional near-infrared spectroscopy (fNIRS) based on craniocerebral correlations. Isolated functional activation was evoked in the motor cortex by a periodically performed finger-tapping task. During 44-channel fNIRS recording, 12 subjects performed 30 trials of right and left index finger tapping in two sessions. The retest interval was set to 3 weeks. Simple correlations of the contrast t-values supplemented by scatterplots, channel-wise intraclass correlation coefficients (ICC), as well as reproducibility indices for the size and the location of the detected activation were calculated. The results at the group level showed sufficient single measure ICCs (up to 0.80) and excellent reproducibility of the size and the location (up to 89% were reproducible). Comparisons of the intersession group amplitudes demonstrate that the fNIRS signals were stable across time in a retest study design: the number of significant differences was less than randomly occurring false-positive activated channels if an alpha level of 5% is chosen. Effect size analyses indicated that the intersession amplitude differences are small (mean < 0.25). For deoxyhemoglobin and oxyhemoglobin distinct statistical power profiles were revealed regarding the activation vs. baseline contrast as well as the intersession amplitude differences, indicating a higher sensitivity of deoxyhemoglobin for local hemodynamic changes. The results suggest that sensorimotor activation assessed by event-related fNIRS based on craniocerebral correlations is sufficiently reproducible at the group level.

Intraobserver Reliability of 4 Physiologic Movements of the Shoulder in Subjects With and Without Symptoms

Valentine RE, Lewis JS. Intraobserver reliability of 4 physiologic movements of the shoulder in subjects with and without symptoms. Objective To assess intraobserver reliability of 4 physiologic movements of the shoulder. Design Test-retest analyses. Blinded data entry. Setting Outpatient department in National Health Service teaching hospital. Participants Forty-five asymptomatic volunteers and 45 subjects with shoulder symptoms. Interventions Not applicable. Main Outcome Measures Intraclass correlation coefficients (ICC), 95% confidence intervals, and standard error (SE) of measurements for bilateral measurements of shoulder flexion and abduction (gravity dependent inclinometer), shoulder external rotation (tape measure), and shoulder internal rotation (visual estimation). Results For subjects without symptoms, single measure ICC results ranged from .85 to .96; SE of measurement results for the angular movements ranged from 2.1° to 2.8° and for the linear measurements 1.1 to 1.6cm. For subjects with symptoms, single measure ICC results ranged from .82 to .98; SE of measurement results for the angular movements ranged from 1.5° to 13.3° and for the linear measurements 1.3 to 1.6cm. Conclusions With the exception of painful shoulder flexion in the group of subjects with symptoms, the single-measure ICC results were very good to excellent and the highest SE of measurement values were 5.3° for the angular measurements and 1.6cm for the linear measurements. For clinicians involved in the management of subjects with shoulder symptoms, the SE of measurement results provide guidance as to the error associated with the individual measurements. Using the SE of measurement results, a clinician may determine if a clinically important change, be it negative or positive, has occurred as a result of any intervention offered.

NIKE: a new clinical tool for establishing levels of indications for cataract surgery

The purpose of this study was to construct a new clinical tool for establishing levels of indications for cataract surgery, and to validate this tool. Teams from nine eye clinics reached an agreement about the need to develop a clinical tool for setting levels of indications for cataract surgery and about the items that should be included in the tool. The tool was to be called 'NIKE' (Nationell Indikationsmodell för Kataraktextraktion). The Canadian Cataract Priority Criteria Tool served as a model for the NIKE tool, which was modified for Swedish conditions. Items included in the tool were visual acuity of both eyes, patients' perceived difficulties in day-to-day life, cataract symptoms, the ability to live independently, and medical/ophthalmic reasons for surgery. The tool was validated and tested in 343 cataract surgery patients. Validity, stability and reliability were tested and the outcome of surgery was studied in relation to the indication setting. Four indication groups (IGs) were suggested. The group with the greatest indications for surgery was named group 1 and that with the lowest, group 4. Validity was proved to be good. Surgery had the greatest impact on the group with the highest indications for surgery. Test-retest reliability test and interexaminer tests of indication settings showed statistically significant intraclass correlations (intraclass correlation coefficients [ICCs] 0.526 and 0.923, respectively). A new clinical tool for indication setting in cataract surgery is presented. This tool, the NIKE, takes into account both visual acuity and the patient's perceived problems in day-to-day life because of cataract. The tool seems to be stable and reliable and neutral towards different examiners.

The NEO-FFI is a reliable measure of premorbid personality in patients with probable Alzheimer's disease

To assess the inter-informant reliability, intra-informant reliability and internal consistency of the NEO-FFI as a measure of premorbid personality in patients with Alzheimer's Disease (AD). One hundred and five persons with NINCDS-ADRDA probable AD for the assessment of inter-informant reliability and internal consistency, and 30 for the assessment of intra-informant reliability. Premorbid personality was rated retrospectively by close relatives remembering the patient as he/she had been when aged in his/her forties. One hundred and five AD patients were rated by two separate informants. Thirty AD patients were rated by the same informant on separate occasions one year apart. Inter-informant reliability for the five domain scores of the NEO-FFI was shown to range from fair to good when measured using the single measure Intraclass Correlation Co-efficient (ICC) (0.52-0.64), and to range from good to excellent when measured using the average ICC (0.68-0.78). Intra-informant reliability for four out of the five domains was shown to be excellent when measured using the single ICC (0.81-0.92), and good for the remaining domain (0.72). Intra-informant reliability was found to be excellent for all five domains when measured using the average ICC (0.84-0.96). Internal consistency of the five domains was good. The NEO-FFI can be used reliably to measure premorbid personality in patients with probable AD. It may be useful to maximise reliability by using a mean domain score based on questionnaires completed by two or more informants who knew the patient well earlier in life.

Event-related functional near-infrared spectroscopy (fNIRS): Are the measurements reliable?

The purpose of the present study was to investigate the retest reliability of event-related functional near-infrared spectroscopy (fNIRS). Therefore, isolated functional activation was evoked in the occipital cortex by a periodic checkerboard stimulation. During a 52-channel fNIRS recording, 12 subjects underwent 60 trials of visual stimulation in two sessions. The retest interval was set to 3 weeks. Linear correlations of the contrast t values supplemented by scatter plots, channel-wise intraclass correlation coefficients (ICC) as well as reproducibility indices for the quantity of activated channels ( R QUANTITY) and the location ( R OVERLAP) of the detected activation were calculated. The results at the group level showed good reliability in terms of the single measure ICCs (up to 0.84) and excellent reproducibility quantified by R QUANTITY and R OVERLAP (up to 96% of the quantity and the location were reproducible), whereas the results at the single subjects' level were mediocre. Furthermore, the reliability assessed by single measurement ICCs improved if regarded at a cluster level.

Comparison of Two Clinical Bleb Grading Systems

To evaluate 2 recently described grading systems for clinical grading of filtering surgery blebs: the Moorfields Bleb Grading System (MBGS) and the Indiana Bleb Appearance Grading Scale (IBAGS). Observational comparative study. Twenty-four glaucoma filtering blebs in 24 eyes of 17 patients. Three observers in a prospective agreement study compared MBGS with IBAGS during slit-lamp examination. Comparison analyses were performed, including agreement, repeatability, and intraclass correlation coefficient (ICC). A wide range of bleb characteristics was represented in the cohort. Acceptable levels of intrasystem agreement were found in both systems: for IBAGS, overall agreement at the 0.5- and 1.0-unit levels were 80.6% and 97.6%, respectively, and for MBGS, these were 78.4% and 97.4% for morphologic and vascularity indices. Repeatability coefficients ranged from 0.5 to 1.4 for MBGS and 0.8 to 1.2 for IBAGS. The ICC values in the MBGS ranged from 0.18 to 0.72 for single measures and 0.39 to 0.88 for average measures. For IBAGS, the single-measure ICC values were between 0.06 and 0.53, and the average-measure ICC values were between 0.16 and 0.77. The MBGS ICC values for bleb size were higher than for IBAGS. Both methods are reproducible clinically and had generally high levels of interobserver agreement. Both have minor deficiencies that should be amenable to improvement. The MBGS performed similarly to the IBAGS for reproducibility, had higher ICC values for morphologic features, and captured extra vascularity data with probable clinical implications.

The reliability of peer review in anthrozoology

We evaluated the reliability of peer review ratings of 218 proposals that were submitted for presentation at the 2004 meeting of the International Association of Human–Animal Interaction Organizations (IAHAIO). Each proposal was evaluated on four dimensions by different sets of three raters who were blind to the identity of the authors. Single measure intraclass correlations between the evaluations of raters indicated that the reliabilities of judgments were poor for abstracts submitted as research presentations and as descriptions of applied human–animal programs, and fair for abstracts submitted as critical reviews. Reliability of the selection system was considerably improved by averaging the scores of the three raters and using the combined scores to make accept/reject decisions. We suggest modifications to the review process that may enhance the reliability of the selection process for proposals submitted for meetings and for journal submissions in anthrozoology.

Reflex vasoconstrictor responses of the healthy human fingertip skin. Normal range, repeatability, and influencing factors

Skin vasoconstrictor reflexes (SVCRs) are mediated by sympathetic vasoconstrictor nerve fibers and used to test sympathetic function of the extremities. Since there is a dearth of published systematic data on inspiratory gasp-induced SVCR, we studied its range, repeatability, and influencing factors in 64 healthy subjects (mean age 41.9 years, range 20–78 years, 32 males). Mean vasoconstriction for the second digit was 54% (±19)/54% (±19) (right/left) and 69% (±16)/68% (±15) for the fourth digit. Forth digit vasoconstriction was significantly larger than the second digit ( P < 0.01), but second and fourth digit correlation was good (Pearson: 0.521–0.741). Repeatability using single measure intraclass correlation was good with 0.62/0.69 for the second digit and 0.64/0.80 for the fourth digit (right/left). A trend for decreasing SVCR was seen with increasing age and weight, no effect was seen for gender, height, or blood pressure. The SVCR repeatability and range of variation seen in this study compare favorably with tests of heart rate variability, thermal threshold detection, and motor nerve conduction.