Criterion Referenced Measures of Technical Proficiency in Maintenance Activities

Abstract

The paper concerns instruments used in the determination of how effeciently maintenance men perform the various tasks of their jobs. Currently, a great deal of reliance is placed on unvalidated job-knowledge tests, theory tests, school marks, supervisors' ratings for such determination. This paper presents a composite of the results of many research and development efforts concerning the empirical validity of such instruments. These results indicate that these measurement instruments have low empirical validity. The limitations of traditional systems effectiveness measures are also discussed. Although job task performance tests (JTPT) have much higher empirical validity, such tests have had very limited use because of their relatively high costs in time, personnel and equipment. As a result of the foregoing findings, the Air Force Human Resources Laboratory (AFHRL) supported a series of efforts to develop better criterion referenced JTPT, and to attempt the development of paper and pencil symbolic substitute tests of high empirical validity. The paper describes the model battery of 48 criterion referenced JTPT, which has been developed to cover all key maintenance activities such as checkout, align/adjust, remove/replace, troubleshooting, test equipment and soldering. During this development many factors were considered including the identification and classification of tasks to be measured, the hierarchal relationship of maintenance tasks, the most effective order of their measurement and the ease of test administration. This battery was developed as a model of JTPT to be used on the job and in training. It was also intended as a battery of criterion tests for the validation of paper and pencil symbolic substitute tests. Batteries of graphic and video symbolic substitute tests were developed and given limited validations. The validation of graphic symbolic substitute tests indicated that the symbolics for all activities, with the exception of soldering, have promise. The paper discusses the requirements for additional refinement and validation for the various graphic tests. An unsuccessful effort to develop video symbolic substitute tests is also mentioned. Suggestions are made for the application of criterion referenced JTPT for the improvement of maintenance efficiency.