Determining Significance Levels in Pattern Analysis

Abstract

The method of Pattern Analysis (Rimoldi & Grib, 1959, 1960) was used to compare the performance of experts, and junior and senior medical students in the Test of Diagnostic Skills (Rimoldi, 1961). This test requires S to select freely as Inany questions as he wants in order to reach a solution ro a diagnoscic problem. Indexes of agreement (Rimoldi & Grib, 1959, 1960) were computed for the patterns of responses of each of the 91 juniors and 143 seniors in relation to the experts' patrern. Also, the index of agreement was determined for the group of juniors and for the group of seniors in relation to the group of experts. The values are, respectively, ,7195 and ,7452 indicating, as hypothesized, that the seniors' patcern agreed more with the experts' pattern than did the juniors' patcern. The frequency distribution of the index of agreement for each individual for both juniors and seniors approximated [he normal curve [X 13.52 (10 df) and X' = 15.56 (9 df), respectively]. The means of these distributions are ,7238 and ,7515 for juniors and seniors which are significantly differenr at P = .013 for a one-railed test. The significance of each S's performance can be determined by using the hypergeometric distribution. The problem consists essentially in determining the probabiliry chat, when either a junior or a senior selects a certain number of questions, k or more of these will agree with questions selected by the experts. This method can also be used to determine test-retest reliability. However, attention is focused not on total number of correct responses, but on the stability of each item. Using the method of Pattern Analysis, reliability will be high only when each S answers each item in the same way in both administrations.