Retrospective Pretest Research Articles

A Comparison of Three Retrospective Self-reporting Methods of Measuring Change in Instructional Practice

In the post + retrospective pretest method of measuring change, evaluators ask the respondents to recall pre-intervention status at posttest time. Research has produced strong evidence in support of this approach over the pretest-posttest approach to measuring change. However, no research has yet to examine and compare different forms of retrospective methods. We compared three retrospective methods of measuring elementary grade teachers’ self-reported change in mathematics instructional practices: the post + retrospective pretest method (reporting current practices and earlier practices), the post + perceived change method (reporting current practice and the amount and direction of change), and the perceived change method (reporting only the amount and direction of change). Teachers in the post + retrospective pretest condition reported least change, followed by teachers in the post + perceived change condition; teachers in the perceived change condition reported the greatest change. We can explain our findings in terms of differential satisficing (the tendency to exert minimal effort in responding) caused by differences in cognitive demands among the three methods. Greater task difficulty leads to greater satisficing, which causes respondents to resort more to socially desirable responses. A greater tendency to provide socially desirable responses leads to relying on expected implicit theory of change and subsequently reporting greater change in instructional practices.

A Comparison of Three Retrospective Self-reporting Methods of Measuring Change in Instructional Practice

In the post+retrospective pretest method of measuring change, evaluators ask the respondents to recall pre-intervention status at posttest time. Research has produced strong evidence in support of this approach over the pretest–posttest approach to measuring change. However, no research has yet to examine and compare different forms of retrospective methods. We compared three retrospective methods of measuring elementary grade teachers’ self-reported change in mathematics instructional practices: the post+retrospective pretest method (reporting current practices and earlier practices), the post+perceived change method (reporting current practice and the amount and direction of change), and the perceived change method (reporting only the amount and direction of change). Teachers in the post+retrospective pretest condition reported least change, followed by teachers in the post+perceived change condition; teachers in the perceived change condition reported the greatest change. We can explain our findings in terms of differential satisficing (the tendency to exert minimal effort in responding) caused by differences in cognitive demands among the three methods. Greater task difficulty leads to greater satisficing, which causes respondents to resort more to socially desirable responses. A greater tendency to provide socially desirable responses leads to relying on expected implicit theory of change and subsequently reporting greater change in instructional practices.

Quality of life in children with otitis media: validity of a survey

Introduction. Validity research into the OM-6 for a Dutch population, quality of life after grommets1 and the influence of ‘response shift bias’ was undertaken. Methods. Parents of 77 children with otitis media (OME) (age 12–36 months) completed the OM-6 (six items, seven-point scale). Quality of life domains concern physical suffering, hearing loss, speech impairment, emotional distress, activity limitations and caregiver concerns. The ‘global quality of life’ rating is performed on a 10-point visual analogue scale. The OM-6 is completed after the first visit to the ear, nose and throat (ENT) surgeon (pre-test 1) and around the time of surgery (pre-test 2). The post-test and the retrospective pre-test are scheduled 6 weeks after the intervention. Results. ‘Global quality of life’ and OM-6 ratings show a good correlation (R = −0.77, P < 0.01). Internal consistency of the OM-6 is satisfying (α = 0.79), responsiveness to clinical change is large (standardized response mean >0.8). Response shift bias is expressed by a difference between pre-test 1 and the retrospective pre-test (T = −3.3, P < 0.01). Before surgery, parents underestimate the seriousness of hearing loss (Z = −3.3, P < 0.05) and overestimate quality of life in general (Z = −3.6, P < 0.05). Conclusion. The OM-6 is valid for a Dutch population. Reported quality of life improves as a consequence of the intervention. The presence of response shift could be explained through adaptation to symptoms before surgery and changes in symptoms as a result of surgery, as well as an increase in knowledge of the consequences related to OME and hearing loss.

Measuring Program Outcomes: Using Retrospective Pretest Methodology

This study used longitudinal data from 307 mothers with firstborn infants participating in a home-visitation, child-abuse prevention program. A self-report measure of specific constructs the program hoped to affect showed that the retrospective pretest methodology produced a more legitimate assessment of program outcomes than did the traditional pretest-posttest methodology. Results showed that when response shift bias was present, traditional pretest-posttest comparisons resulted in an underestimation of program effects that could easily be avoided by the retrospective pretest methodology. With demands for documenting program outcomes increasing, retrospective pretest designs are shown to be a simple, convenient, and expeditious method for assessing program effects in responsive interventions. The limits of retrospective pretests, and methods for strengthening their use, are discussed.

Measuring program outcomes: using retrospective pretest methodology

This study used longitudinal data from 307 mothers with firstborn infants participating in a home-visitation, child-abuse prevention program. A self-report measure of specific constructs the program hoped to affect showed that the retrospective pretest methodology produced a more legitimate assessment of program outcomes than did the traditional pretest-posttest methodology. Results showed that when response shift bias was present, traditional pretest-posttest comparisons resulted in an underestimation of program effects that could easily be avoided by the retrospective pretest methodology. With demands for documenting program outcomes increasing, retrospective pretest designs are shown to be a simple, convenient, and expeditious method for assessing program effects in responsive interventions. The limits of retrospective pretests, and methods for strengthening their use, are discussed.

Estimating the Impact of Health and Safety Training Using the Retrospective Pretest Design

Abstract An alternative to the traditional pretest/postest method of assessing training program effectiveness from hazardous waste worker training, called the retrospective pretest design, is explored in this study. In the retrospective pretest design, trainee self-reports about their knowledge, skills, abilities, and/or attitudes are obtained only after the completion of the training, not before. Trainees provide two types of self-reports: (1) the traditional posttraining outcome responses and (2) responses on how they would have answered those same outcome items before they participated in the health and safety program. These latter responses are known as retrospective pretests, and comparison of the two responses for an item is considered a better measure of true change due to training than the traditional pretraining and posttraining responses to the same item. Data from five types of hazardous waste worker health and safety training curricula (40-hour site worker; 24-hour treatment, storage, and disp...

The Effect of Occupational Therapy Education on Students’ Perceived Attitudes Toward Persons With Disabilities

The effect of occupational therapy education on students' perceived attitudes toward persons with disabilities was studied. The perceived attitudes of 144 occupational therapy students toward persons with disabilities were measured before (retrospective pretest) and after (posttest) the students attended formal professional education at the University of Alberta. Posttest scores were significantly higher than the retrospective pretest scores, indicating that students' attitudes became more positive after they commenced formal professional education. The posttest scores of students at various levels of education, however, did not show any significant difference. On the basis of these observations, a positive but nonlinear relationship between occupational therapy education and attitudes toward persons with disabilities was postulated.

Implications of the Pre/Post/Then Design for Evaluating Social Group Work

Pre/post designs that rely on self-report measures are potentially contaminated by response shift bias, a change in respondents' understanding of the phenomena being measured between the pre- and posttests. Retrospective pretests, or then tests, have been shown to be useful in identifying response shifts. Pre/post/then designs are highly utilitarian designs that have the potential of providing more accurate assessments of respondent change. This article reviews the literature on response shift bias and the utility of retrospective pretests m detecting such bias. Three studies that used the pre/post/then design to evaluate self-reported changes in caseworker abilities (knowledge or skill) relative to practice evaluation, basic interviewing skills, and/or assessment are described.

Evaluation of an intervention for staff in a long-term care facility using a retrospective pretest design.

It has been previously established that human service workers often suffer from emotional exhaustion, which has been conceptualized by Maslach and Jackson as burnout. Burnout may be a particularly great risk in workers providing long-term geriatric care. The current study evaluated the effects of a series of three 3-hour sessions designed to address team building, communication skills, self-esteem, and stress management on a random sample of 51 of the 188 long-term care staff who participated. Using a retrospective pretest design, a statistically significant improvement from "then" to "today" was found for the three components of burnout: Depersonalization, Emotional Exhaustion, and Personal Accomplishment. Responses to an open-ended question about workshop effects corroborated the quantitative data, and effects noted were highly related to the defined objectives of the workshops.

The Retrospective Pretest and the Role of Pretest Information in Evaluative Studies

To test the effect of training programs, subjects are often given self-report questionnaires. When such self-report instruments are used, response-shift bias is a possible confounder of results. According to the response-shift theory, pretest information given to subjects might reduce or diminish response-shift bias. In this experiment pretest information was given prior to a course in leadership. Contrary to expectation, the subjects' ratings were somewhat lower in the pretest-information condition. It was concluded that the pretest information provided was not effective enough. Use of a correct amount of reliable pretest information is recommended, otherwise this information may function as a ‘minicourse’ in itself and thereby confound the hypothesized or expected results.

Invalidity of True Experiments

The validity of true experiments is threatened by a class of self-report biases that affect all respondents at pretest but which are diminished by treatment, yielding noncomparable treated and control subjects at posttest. These biases include inaccurate self-evaluations due to (a) lack of understanding of dimensions of self-rating, (b) unconscious needs to rationalize sources of severe emotional distress, (c) distressed or altered states (e.g., drug states) that lower the ability to report accurately, and (d) deliberate impression management in service of accessing desired treatment. They are detectable via external criteria, special conditions of measurement, and retrospective pretests, and may be lessened in several ways. Unchecked, they produce treatment by self-report bias interactions that undermine the validity of even true experiments.

The Comparison Group Problem in Bereavement Studies and the Retrospective Pretest

An elaboration of Campbell and Stanley's (1963) retrospective pretest design to include a "surrogate comparison group" is described. The successful use of the design is illustrated and tested using data from a longitudinal investigation of bereavement outcome following pregnancy loss. The bereaved group was not significantly different from the surrogate comparison group of currently pregnant women in their retrospection of their prepregnancy mental health status. The pregnant surrogate group's current mental health status also did not differ from the bereaved group's retrospections concerning their mental health during their pregnancies. Implications for the use of the design are also discussed.

Pretesting effects in retrospective pretest^posttest designs.

Two studies evaluated communication skills training by using a pretest-posttest design, including retrospective pretest ratings, to control for response shift bias. A response shift is a change in a subject's internal standard for determining his or her level of functioning on a given dimension. In Exp 1, Ss were 37 hospital employees. Data indicated that the self-report pretest exerted a clear effect on subsequent self-report posttest and retrospective pretest ratings. Training was ineffective and a response shift did not occur. Experimental Ss could not remember and control Ss could remember their pretreatment ratings to a reasonable extent. In Exp 2, Ss were 58 3rd-year dental students. Results show that the training was effective. A behavioral pretest administered prior to the self-report pretest prevented a response shift from occurring. This finding gives empirical support to the contention that Ss' lack of sufficient information about their level of functioning at pretest may be a causal determinant of the response shift. Data also indicate that the retrospective pretest is robust for procedural differences in administering this instrument.

Subject bias and the retrospective pretest in retrospect

Studies employing designs including a thentest were reviewed to examine to what extent they demonstrated a response shift, while controlling for subject bias by including a placebo control condition. A computer search was executed to detect articles that included the Howard, Ralph, et al. (1979) article in their reference list. Of 24 studies, 16 did not warrant any statement about the occurrence of a response-shift bias, relative to a subject bias, because they lacked a placebo control condition. Support for the subject bias hypothesis, relative to the response-shift hypothesis, was found in one of eight studies that did include a placebo control condition. With respect to these results, three supplementary considerations were taken into account. It was concluded that subject bias has not proved to be an unqualified alternative explanation of the response-shift interpretation in the reported studies.

Pretesting effects in retrospective pretest-posttest designs.

Pretesting effects in retrospective pretest-posttest designs.

Retrospective Measures in the Modification of Attitudes Toward Persons with Disabilities

Many rehabilitation studies which attempt to determine the effectiveness of procedures designed to modify attitudes toward persons with disabilities rely on self-report measures. When utilizing self-report instruments within a pretest/posttest design, a potential threat to internal validity is the shift of response interpretation on the part of the subject. One way of dealing with this response-shift is to utilize retrospective pretests in addition to a pre-treatment pretest. Two studies are presented which compared the use of traditional pretests with retrospective pretests in the modification of attitudes toward the disabled. Both studies showed that the retrospective pretest design was more sensitive to perceived attitudinal changes.

A method for analyzing retrospective pretest/posttest designs: I. Theory

To control for response-shift bias in pretest/posttest designs, Howard and his collaborators proposed the use of retrospective pretests, the so-called thentests. If a response-shift bias is present, thentest/posttest difference scores will yield better estimates of a treatment effect than will the usual pretest/posttest difference scores. In this article a method is presented that is aimed at detecting response-shift bias in retrospective pretest/posttest designs. This method, essentially a hierarchical model fitting procedure, is easy to apply, prevents capitalization on chance, and makes optimal use of information supplied by the control group.

The retrospective pretest: an alternative approach in evaluating drug prevention programs.

Two self-report techniques assessed student behaviors before and after a life-skills substance abuse prevention program, targeted at seventh graders in an inner-city elementary school. The retrospective pretest/posttest technique demonstrated no significant changes in drug usage, while a traditional pretest/posttest indicated significant increases in tobacco usage. The findings of this study support the continued use and research on the retrospective pretest, particularly as they relate to tobacco usage. Given participants' inhibitions to self-report deviance, inclusion of this technique can provide a more complete assessment of prevention programs.

Response-Style Effects, Response-Shift Bias and a Bogus-Pipeline

The influence of response-style effects on subjects' self-ratings was assessed, utilizing a bogus-pipeline technique. The design was a pretest-posttest design, including retrospective preratings. Subjects were 73 psychology freshmen fulfilling a course requirement. Subjects were led to believe that the veracity of their self-reports could be checked by means of objective measures. The bogus-pipeline induction in the pretesting did lower self-reported preratings and consequently eliminated response-shift bias, defined as a significant mean difference between conventional and retrospective preratings. The results show no contradiction of earlier research. Since a response shift occurred in the experimental nonbogus-pipeline condition, it was concluded that use of a bogus-pipeline technique can improve self-reported pretest scores and subsequently eliminate response-shift-bias effects. Data furthermore show that the retrospective pretest is rather robust for procedural differences.

Self-Report Evaluations of Training Effectiveness: Measuring Alpha, Beta, and Gamma Change

Although self-report measures are often used as criteria in treatment interventions, the contaminating effects of response-shift bias associated with these scales are less frequently evaluated. G.S. Howard and his co-workers (Howard, Schmeck, &amp; Bray, 1979) have suggested that response-shift bias can be measured and controlled by gathering and analysing retrospective Pretest scores in addition to the conventional Pretest and Posttest information normally obtained in change studies. In this study these measures were gathered from participants in an assertiveness training programme. The aim was to assess the extent to which subjects had recalibrated the scales of measurement (i.e., beta change) and reconceptualized the construct of assertiveness (i.e., gamma change) from Pretest to Posttest. These two forms of response-shift bias were recorded for a number of trainees. Apart from finding that the usual Pretest/Posttest design offers a more conservative estimate of treatment effects than the approach using retrospective scores, the results showed that the latter approach has particular utility in diagnosing the nature and extent of change experienced by individuals in treatment interventions.