Reliable Instrument For Evaluation Research Articles

Mentoring Faculty to Improve Teaching and Student Learning

For teaching to improve, three factors must be in place. First, the college must use a valid and reliable student evaluation instrument in combination with other feedback mechanisms. Second, there must be an accompanying norming report that provides comparative and diagnostic information to departmental chairs and teachers. Raw data from even a valid and reliable instrument may not be informative. Third, the college must have a mentoring process in place. This paper discusses briefly each of the three elements to initiate and sustain a successful mentoring program to improve teaching and student learning.

Psychometric Characteristics of the Spanish Version of the FAB Questionnaire

Validation of a translated, culturally adapted questionnaire. To translate and culturally adapt the Spanish version of the Fear Avoidance Beliefs Questionnaire (FABQ), and to validate its use in Spanish-speaking patients with low back pain (LBP). The FABQ is a reliable evaluation instrument for fear avoidance beliefs, which includes two subscales (FAB-Work and FAB-Phys). No validated Spanish version was available. Translation/retranslation of the English version of the FABQ was done blindly and independently by a multidisciplinary team. The study was done in 12 primary care centers and 9 hospital outpatient clinics from seven regions in Spain, with 209 acute, subacute, and chronic patients who visited their physician for LBP: 53 in the pilot phase and 156 in the validation phase. Subjects were given the FABQ, two VAS for LBP and referred pain, and the Roland-Morris and SF-12 Questionnaires on their first visit and 14 days later. In the pilot phase, on day 1 test-retest reliability was estimated by giving a second FABQ in which the name and order of the items had been changed. Time necessary to complete the FABQ was [median, P25, P75] 10 minutes (5,15). FABQ values were not normally distributed. Comprehensibility: No request for aid in interpretation was made during the validation phase and no item was left unanswered by >or= 10% of patients. Reliability: Scores [median, P25, P75] of the two FABQ were: 72.00 (47.25, 82.00) and 72.00 (49.50, 83.75), with an intraclass correlation coefficient of 0.9668 (95% confidence interval, 0.9421,0.9823). Mean of kappa values for all items was 0.743. Internal Consistency: Cronbach's alpha was 0.9337. Values of FAB and FAB-Work were >37% higher for patients on sick leave. For FAB-Phys differences were below 8%. FABQ, FAB-Work and FAB-Phys strongly correlated with disability on days 1 and 15. The Spanish version of the FABQ has good comprehensibility, internal consistency, and reliability. It cannot be analyzed parametrically, but only nonparametrically. The total FABQ is at least as valid as its subscales and simpler to score, making it more suitable for routine clinical use.

Mentoring Faculty to Improve Teaching and Student Learning

ABSTRACTFor teaching to improve, three factors must be in place. First, the college must use a valid and reliable student evaluation instrument in combination with other feedback mechanisms. Second, there must be an accompanying norming report that provides comparative and diagnostic information to departmental chairs and teachers. Raw data from even a valid and reliable instrument may not be informative. Third, the college must have a mentoring process in place. This article discusses briefly each of the three elements to initiate and sustain a successful mentoring program to improve teaching and student learning. Of course, if the school does not reward improved or effective teaching it is not likely to occur.

Validation of Impact of Events Scale in nurses under threat of contagion by severe acute respiratory syndrome

The purpose of the present study was to validate the Impact of Events Scale (IES) in a sample of nurses working under threat of severe acute respiratory syndrome (SARS). The internal consistency, construct validity and convergent validity of the instrument were examined in a sample of 128 nurses during the SARS outbreak in Taiwan. Principal component analysis followed by a quartimax rotation were used to derive a two-factor solution, labeled intrusion (factor 1) and avoidance (factor 2), with both factors accounting for 50.7% of the explained variance. The total Cronbach's alpha of 0.90 reflected the good internal consistency of the instrument. Correlation with the Symptoms Checklist 90-R demonstrated the convergent validity of the IES. In conclusion, the IES can be used as a convenient, reliable and valid instrument for evaluation of the psychological distress of nurses working with the highly contagious disease.

Clock Drawing Test-Modified and Integrated Approach (CDT-MIA): description and preliminary examination of its validity and reliability in dementia patients referred to a specialized psychogeriatric setting.

The authors developed a scoring system for clock drawing, based on modification and integration of 3 established scoring methods. The Clock Drawing Test-Modified and Integrated Approach (CDT-MIA) is a 4-step, 20-item instrument, with a maximum score of 33, which emphasizes differential scoring of contour, numbers, hands, and center. It was administered to 139 patients (93 with and 46 without dementia). Dementia patients revealed significantly more impairment on the CDT-MIA total score and hours and hands subscores. Correlations between CDT-MIA and 2 CDTs were high. With receiver operating characteristics (ROC) curves, the area constructed under CDT-MIA curve was large. The best trade-off between sensitivity and specificity for CDT-MIA was the cut-point 23 (91% and 80%, respectively). The internal consistency of CDT-MIA was high, and there was a high degree of interrater reliability. Thus, CDT-MIA was found to be a valid and reliable evaluation instrument for dementia patients in a specialized setting.

Validation of the Tunisian version of the Roland-Morris questionnaire.

Our aim was to validate a culturally adapted, Tunisian-language version of the Roland-Morris Disability Questionnaire (RMDQ), which is a reliable evaluation instrument for low-back-pain disability. A total of 62 patients with low back pain were assessed by the questionnaire. Reliability for the 1-week test/re-test was assessed by a construction of a Bland Altman plot. Internal construct validity was assessed by Cronbach's alphatest. External construct validity was assessed by association with pain, the Schober test and the General Function Score. Sensitivity to change was determined using a t-test for paired data to compare RMDQ scores at inclusion and at completion of the therapeutic sequence of local corticosteroid injections. We also compared the questionnaire score with the General Function Score, both taken after completion of the therapeutic sequence. The constructed Bland Altman plot showed good reliability. Internal consistency of the RMDQ was found to be very good and the Cronbach's alpha test was 0.94, indicating a good internal construct validity. The questionnaire is correlated with the pain visual analogue scale (r=33; p=0.0001), with the Schober test (r=0.27; p=0.0001) and the General Function Score (r=56; p=0.0001) indicating an adequate external construct validity. The RMDQ administered after the therapeutic sequence is sensitive to change (r=0.83; p=0.000). Comparison of the questionnaire score to the General Function Score, after completion of the therapeutic sequence, was satisfactory (r=0.75; p=0.000). We conclude that the Tunisian version of the Roland-Morris questionnaire has good reliability and internal consistency. Furthermore, it has a good internal- and external construct validity and high sensitivity to change. It is an adequate and useful tool for assessing low-back-pain disability.

Validation of the spanish version of the Roland-Morris questionnaire.

Validation of a translated, culturally adapted questionnaire. To translate and culturally adapt the Spanish version of the Roland-Morris Questionnaire (RMQ), and to validate its use for assessing disability in Spanish patients with low back pain (LBP). The RMQ is a reliable evaluation instrument for disability, but no validated Spanish version is available. Translation/retranslation of the English version of the RMQ was done blindly and independently by four different individuals, and adapted by a team that included five primary care physicians, three back specialists, and two methodologists. The study was done in the primary care setting in Mallorca, with 195 patients who visited their physician for acute or chronic LBP: 50 in the pilot study and 145 in the validation study. Individuals were given the RMQ and three other scales (VAS, Oswestry, and EuroQol) on their first visit and 14 days later. For the pilot study, on the first visit, patients were also given a second RMQ with the questions in a different order and they were subsequently asked for comprehension of each item of the questionnaire. Only 2 questions were partially rephrased after the pilot study, and no request for aid in interpretation was made during the validation study. Scores of the two RMQs on day 1 were 10.04 (SD, 5.01) and 10.9 (SD, 4.8), with an intraclass correlation coefficient of 0.874. The 95% limits of agreement by the Bland-Altman method was 0.340 +/- 4.81. Cronbach's alpha was 0.8375 (day 1) and 0.9140 (day 15) in the validation. Concurrent validity, measured by comparing RMQ responses with the results of VAS, was r = 0.347 (P = 0.0000) for day 1,and r = 0.570 (P = 0.0000) for day 15. Construct validity, tested by determining the correlation between the Spanish RMQ and the Spanish adaptation of the Oswestry Questionnaire, yielded r = 0.197 (P = 0.0061) on day 1 and r = 0.341 (P = 0.0000) on day 15. The Spanish version of the RMQ has good comprehensibility, internal consistency, and reliability, and is an adequate and useful instrument for the assessment of disability caused by LBP.

Hairdex

Because of a strong reduction of life quality in patients with hair loss, a special questionnaire was developed following the general Skindex questionnaire for dermatoses. The Hairdex was evaluated in 75 female hair patients. The aim of the study was to prove the questionnaire's statistical values, reliability, validity, general acceptance and accuracy. The clinical manifestation of hair loss was categorized as "not visible", slightly visible" and "obviously visible". The hair loss lead to great differences in the life quality of both the patients with obvious hair loss and in patients with non-visible hair loss, especially in the categories "emotions", "self-confidence" and "stigmatization". The convergent and discriminant validity of the questionnaire was satisfactory. The acceptance of the questionnaire was very good with 90%. The hairdex-questionnaire represents a reliable and patient-oriented instrument for evaluation of life quality in hair diseases. The reliability of the questionnaire concerning longitudinal therapeutic effects will need to be investigated in further studies.

Hand mobility in scleroderma (HAMIS) test: The reliability of a novel hand function test

Hand Mobility in Scleroderma (HAMIS) is a new hand function test developed for adults who have systemic sclerosis. HAMIS consists of 9 items designed to measure all movements assessed in an ordinary range of motion (ROM)-measured hand test. The aim of this study was to examine the reliability of the test. Two observers (one occupational therapist and one physiotherapist) performed the assessments independently of each other on 30 adult subjects. There were 25 women and 5 men in the sample (mean age 53 years, average time since diagnosis 4 years). Subjects were tested twice by each rather. Internal consistency ranged from 0.80 to 0.85 (Cronbach's alpha). Agreement between the two observers was good for all items (estimated kappa 0.52 to 1.00). The agreement between the observers' first and second assessments was moderate to very good for most items (estimated kappa 0.48 to 1.00), but not for the assessment of supination (estimated kappa 0.25 to 0.59). HAMIS is a reliable instrument for evaluation of hand function on scleroderma patients.

Simulations in distance education: Progress towards an evaluation instrument

This article reports on a study in which an interactive computer‐based learning resource consisting of a tutorial with a decision support system and simulations, is evaluated in the context of a distance education business degree course. The primary purpose of the study was to develop and test an instrument for evaluation of the learning resource, with particular reference to simulations. The instrument adds further dimensions to factors such as learner support and engagement, to incorporate elements of motivation and transfer. Consequently, the scales that comprise the instrument include three key areas: features of the computer package, the effect on students” motivation, and the impact on their ability to transfer knowledge to the workplace. As a result of the analysis, a reliable and valid evaluation instrument that includes components of these three areas is provided. It is expected that other researchers will be able to develop and refine this instrument to evaluate various forms of computer‐based educational resources.

Development of the Texas Textbook Evaluation Tool (T-TET).

Nurse educators face many challenges in evaluating textbooks to meet the needs of their students and preparing professional nurses who will practice in the next millennium. Using four phases of development, the Texas Textbook Evaluation Tool (T-TET) was created to develop a reliable and valid textbook evaluation instrument. The T-TET may be used to evaluate all levels of textbooks, including specialty textbooks in nursing.

Reliability of a modified medication appropriateness index in ambulatory older persons.

To evaluate the reliability of a medication appropriateness index (MAI) modified for elderly outpatients in a non-Veterans Affairs setting. Reliability study. General community. Ten community-dwelling elderly (> 65 y) taking five or more regularly scheduled medications and participating in a university-based health service intervention study. Interrater reliability of MAI ratings of 65 medications made by two clinical pharmacists for individual items and for an overall summed score was calculated by use of kappa statistics and intraclass correlation coefficient. The interrater agreement for each of the individual MAI items was high for both appropriate and inappropriate ratings and ranged from 80% to 100% (overall kappa = 0.64). Overall agreement for the summed score was good (intraclass correlation = 0.80). The modified MAI is a reliable instrument for evaluation of medication appropriateness in a non-Veterans Affairs, ambulatory, elderly population and may provide pharmacists with a practical and standard method to evaluate patients' drug regimens and identify some potential drug-related problems.

Evaluating ATM Technology for Distance Education in Library and Information Science

Following the lead of distance education studies from Oklahoma and Hawaii, the present study investigated the impact of asynchronous transfer mode (ATM) technology in an interactive environment providing distance education in library and information science at two San Jose State University (SJSU) sites separated by four hundred miles in California. SJSU is in the southernmost city in the San Francisco Bay Area; the distance site, California State University-Fullerton (FLRTN), is in the greater Los Angeles basin. The major purpose of the study was to develop a reliable and valid evaluation instrument. Graduate students were asked to respond to Likert statements via paper-and-pencil administration. Iterative validation of scale components involved 252 responses over three semesters and yielded six scales with final alpha levels from.84 to .89. These scales were used to evaluate the delivery of five courses in the fall of 1995 (N=112). The comparisons between SJSU and FLRTN showed no significant differences in perceptions regarding site setup, communication, or affective relations; there were significant differences between SJSU and FLRTN in perceptions of instructors, technology, and learning. Comparisons across courses found significant differences between the send vs. receive sites, the sense of community, and the perception of instructors. Analyses of modal item scores found: (1) student reaction was more positive at the originating site, regardless of course (2) students bonded with their distant colleagues, or not, in approximately equal numbers; (3) class discussions are not easy in distance mode; and (4) students believe more face-to-face contact with the instructor would be helpful. By far the most prevalent item score was four on a five-point scale, where five equals positive experiences. Students did not perceive themselves as second-class citizens in this instructional mode when compared with a traditional course arrangement

Using the Early Childhood Environment Rating Scale (ECERS) to evaluate quality of nursery provision in England: some data concerning reliability

SummaryAs the demand for research into quality of day care provision expands, so the need for reliable evaluation instruments becomes more acute. This short report describes data collected from 113 day nurseries using the Early Childhood Environment Rating Scale (ECERS). Reliability analyses suggest that the ECERS is an effective research tool for evaluating specific aspects of quality of nursery provision.

Cancer rehabilitation.

An important trend in cancer treatment is an increasing emphasis on the overall outcome, not just on survival or disease-free interval, but also on the functional status and the quality of life of the survivor. Measuring the quality of life as a part of a clinical protocol is not enough. Measures must be aimed at facilitating optimal function and life satisfaction for the patient treated for cancer. The rehabilitation techniques used to enhance the life of the survivor need to be subjected to the same sort of scientific rigor required for other kinds of treatment. To develop data on rehabilitation parameters, we must create and utilize standardized and reliable evaluation techniques and instruments. These evaluation practices can then be utilized in clinical trials, both to assess the effectiveness of the rehabilitation techniques themselves and also to determine the effect of specific antitumor treatments on the patient's physical and psychosocial function. An organized multidisciplinary cancer rehabilitation program as a part of clinical and research facilities may be helpful. The important concept for all of us involved with the care of cancer patients is that of conveying our concern, not only for their survival, but for their function--physical, emotional, social, and vocational.

Assessing the outcomes of nursing care.

There are many dimensions to outcome measurement for patient care and this study focuses on one aspect of outcome, namely that which is most concerned with the immediate effects of nursing care. Traditionally, outcome studies have been almost exclusively concerned with measuring the clinical outcomes of care with very little attempt being made to assess the effect of the nurses' contributions to that care. One of this study's purposes has been to redress this imbalance. As part of a research project on skill mix and the effectiveness of nursing care, a number of patient-focused standards related to the outcome of nursing care were designed. These were tested, in a case study situation, on 15 wards at seven acute hospitals. Initial testing indicates that these outcome measures show promise as a valid and reliable evaluation instrument with the utility for easy application in the clinical setting. They are being presented as a possible way forward to assessing the outcomes of nursing care.

Patient instructors as teachers and evaluators.

To provide objective and reliable evaluation of students' physical diagnosis skills, nonphysician patient instructors (PIs) with stable findings were trained by physicians to function in the multiple roles of patient, teacher, and evaluator. The PIs were taught physical examination techniques and the significance of each of their positive findings. Each second- and third-year medical student examined a cardiovascular and pulmonary PI. At the completion of each examination the PI evaluated the thoroughness and proficiency of the student's examination technique and ascertained whether the student correctly identified and described the abnormal findings. Objective and reliable evaluation instruments were developed. Data are presented to indicate that PIs can be used to evaluate objectively physical diagnosis skills of medical students.

A Pragmatic Approach to Criticism of Multimedia

Because multimedia presentations are used to communicate diverse ideas, and because they employ various mixtures of media, they have resisted systematic criticism. A functional/experiential approach to analysis and judgment of these programs is recommended. Such a system would emphasize the structure and style of the presentation, its technical and aesthetic factors, its value to the audience, and the critic's personal reactions. This system is intended for “layman” critics and incorporates material from aesthetics, communication, instruction, and persuasion. After proper testing of this theoretical approach, a reliable evaluation instrument will be available for distribution.