Final Report - Identification, Development and Validation of Predictors for Successful Lawyering

Abstract

For years, American law schools have placed very substantial weight on applicants' LSAT scores when making admission decisions. As applicant volume and the perceived importance of law school rankings have grown, that emphasis has intensified. Only a small fraction of graduates of law schools become academics, but more than either other professional or academic graduate schools, law schools rely mainly on cognitive and academic indicators to choose among applicants and predict their success in law school. The unresearched question, however, is whether the prediction of lawyering effectiveness should play a role in schools' choices among qualified applicants. This exploratory empirical study sought to answer that question. Funded in large part by the Law School Admission Council, nine years of research are reported in this working paper. The research plan involved three phases. Phase I investigated what lawyers thought were the characteristics of effective lawyers. Twenty-six Effectiveness Factors were extracted from multiple rounds of interviews and focus groups with Berkeley alumni, faculty, students, judges and clients, who represented a wide range of practice settings and specialties. These 26 include not simply analysis and reasoning but also such factors as negotiation, advocacy, stress management, practical judgment, problem solving and other dimensions. The interviews and focus groups generated approximately 715 examples of behavior to illustrate different levels of effectiveness for each of the factors. In an internet survey, more than 2000 Berkeley alumni rated the effectiveness of the examples on a scale that ranged from poor to excellent. Results of the survey, which took into account agreement among respondents, yielded 26 Effectiveness Factor scales that would allow evaluation of individual attorneys' performance. Phase II of the research examined a range of existing tests and test methods, eventually selecting nine types of tests that we hypothesized might predict effectiveness in the 26 identified factors. We used some off the shelf tests, adapted others and wrote items from scratch for others. In Phase III, Berkeley alumni were recruited as test subjects to take a compilation of the chosen tests, which were administered online and took about 2 hours to complete. Subjects also gave consent for researchers to access their UGPA, LSAT scores, and LSGPA. In addition, they identified two supervisors and two peers, as well as themselves, to evaluate their current work effectiveness using the relevant subset of rating scales developed in Phase I. Eleven hundred alumni volunteers participated, two-thirds from Berkeley Law and about one third from Hastings College of the Law. More than 4000 performance appraisals were submitted online. Results revealed the following: 1) A number of the new tests were predictive of lawyer performance on most of the 26 Effectiveness Factors. The LSAT and Index Scores were not particularly predictive of lawyering performance except on the several most cognitively oriented factors among the 26. 2) The new tests were no more predictive of first year law grades than the currently used Index Score but the sample replicated the results typical of the predictive power of the Index Score. 3). Performance on the new tests did not show significant differences between race and gender sub-groups, although it is widely recognized that existing LSAT and Index scores create substantial adverse impact. 4) The Report strongly recommends that this exploratory research be carried forward through conduct of a national validation study such that these methods could potentially be added to existing methods of applicant assessment.