CONSTRUCT VALIDITY OF FREE‐RESPONSE AND MACHINE‐SCORABLE VERSIONS OF A TEST OF SCIENTIFIC THINKING

Abstract

ABSTRACTThe purpose of the study is to investigate the construct validity of free‐response and machine‐scorable versions of Formulating Hypotheses (FH), one of the Tests of Scientific Thinking. Six free‐response and eight machine‐scorable FH items were administered (in that order) to 172 college seniors who were majoring in psychology and who claimed an intention to go to graduate school. Tests of other variables that are theoretically related to FH scores provided data for studying construct validity; these included tests to identify eight cognitive and three personality factors. A background questionnaire was also administered, and GRE scores were obtained from files.Six FH scores were obtained for each item in each test format. They are (1) quality of the hypothesis considered best by the candidate, (2) average quality of all hypotheses, (3) quality of the best hypotheses according to our scoring system, (4) number of hypotheses, (5) number of unusual hypotheses, and (6) number of hypotheses that are both unusual and of high quality. Correlations between corresponding scores for the two formats were computed and corrected for unreliability; they indicate that the two forms cannot be considered parallel. The correlations are particularly low for scores based on numbers of hypotheses.The major method of analysis employed maximum likelihood factor analysis to verify a factor model involving the construct validity variables and to find the extension loadings of the experimental test scores and GRE scores on the factors. One such analysis used cognitive factors and one examined personality factors.The personality factor model identified three factors; social extraversion, maladjustment (anxiety), and acceptance of complexity. GRE scores and quality scores from FH all were related to acceptance of complexity, and GRE scores were all negatively correlated with social extraversion. Number scores from the free‐response form of FH were related (positively) to maladjustment.The cognitive factor model involved eight factors: verbal comprehension, three reasoning abilities, cognitive flexibility, two divergent production factors, and knowledge of psychology. GRE scores and quality scores from both forms of FH all showed substantial correlations with reasoning, cognitive flexibility, and knowledge. Only the number scores from the free‐response form of FH were related to divergent production.An appraisal of all the findings suggests that the machine‐scorable form of FH would add little to the information already available from GRE scores. The free‐response form would add information about divergent production not now provided by GRE tests.