Accuracy of Four Language Analysis Procedures Performed Automatically

Abstract

Most software for language analysis has relied on an interaction between the metalinguistic skills of a human coder and the calculating ability of the machine to produce reliable results. However, probabilistic parsing algorithms are now capable of highly accurate and completely automatic identification of grammatical word classes. The program Computerized Profiling combines a probabilistic parser with modules customized to produce four clinical grammatical analyses: MLU, LARSP, IPSyn, and DSS. The accuracy of these analyses was assessed on 69 language samples from typically developing, speech-impaired, and language-impaired children, 2 years 6 months to 7 years 10 months. Values obtained with human coding and by the software alone were compared. Results for all four analyses produced automatically were comparable to published data on the manual interrater reliability of these procedures. Clinical decisions based on cutoff scores and productivity data were little affected by the use of automatic rather than human-generated analyses. These findings bode well for future clinical and research use of automatic language analysis software.