Bar Exam Research Articles

The exaggerated benefits of failure.

Commencement speakers, business leaders, and the popular press tell us that failure has at least one benefit: It fuels success. Does it? Across 11 studies, including a field study of medical professionals, predictors overestimated the rate at which people course correct following failure (Studies 1-4). Predictors overestimated the likelihood that professionals who fail a professional exam (e.g., the bar exam, the medical boards) pass a retest (Studies 1a, 1b, and 2a), the likelihood that patients improve their health after a crisis (e.g., heart attack, drug overdose; Studies 2b and 6), and the probability, more generally, of learning from one's mistakes (Studies 3-5). This effect was specific to overestimating success following failure (Study 4) and erasing mention of an initial failure that had actually occurred corrected the problem (Studies 2a and 2b). The success overestimate was due, at least in part, to the belief that people attend to failure more than they do (Studies 5 and 6). Correcting this overestimate had policy implications. Citizens apprised of the sobering true rate of postfailure success increased their support for rehabilitative initiatives aimed at helping struggling populations (e.g., people with addiction, ex-convicts) learn from past mistakes (Studies 7a-7c). (PsycInfo Database Record (c) 2024 APA, all rights reserved).

2023년 헌법 중요 판례 평석 (2)

This paper analyzed the meaning and content of major decisions made by the Constitutional Court in 2023 except for normative control. The Constitutional Court determined that the inspection of some items conducted by Gyeonggi-do Province violated Namyangju's local autonomy because the inspection was not specified or its relevance to the original inspection was not recognized. The Constitutional Court rejected the request for a competent dispute trial sought by the Minister of Justice and the prosecutor for the revision of the Public Prosecution Service Act and the Criminal Procedure Act by the National Assembly. The court opinions of the five judges denied the eligibility of the Minister of Justice as a claimant and the possibility of infringement of the prosecutor's authority. The Constitutional Court decided on the Ministry of Justice's announcement that prohibited the test of confirmed patients and restricted the test of self-quarantined and high-risk people in the 10th bar exam, which was scheduled to take effect in early January 2021, when COVID-19 was prevalent. The Constitutional Court judged that the uniformly prohibition of the respondent from taking the test due to vague concerns violated the claimants' freedom to choose their jobs. The Constitutional Court determined that the 'right to be registered' as soon as birth was a fundamental right guaranteed by the Constitution. In relation to the Incheon Metropolitan City lawn, the Constitutional Court judged that the Incheon Metropolitan City Ordinance, which restricts the use of the lawn for the purpose of assembly and demonstration, violates the Constitution by violating the claimants' freedom of assembly and association. With a 7:2 opinion, the Constitutional Court ruled that the 'Act on the Development of South and North Korean Relations' clause, which prohibits the spraying of leaflets to North Korea and punishes violations, violates the claimants' freedom of political expression.

Healthcare Violence and the Potential Promises and Harms of Artificial Intelligence.

Currently, the healthcare workplace is one of the most dangerous in the United States. Over a 3-month period in 2022, two nurses were assaulted every hour. Artificial intelligence (AI) has the potential to prevent workplace violence by developing unique patient insights through accessing almost instantly a patient's medical history, past institutional encounters, and possibly even their social media posts. De-escalating dialog can then be formulated, and hot-button topics avoided. AIs can also monitor patients in waiting areas for potential confrontational behavior.Many have concerns implementing AIs in healthcare. AIs are not expected to be 100% accurate, their performance is not compared with a computer but instead measured against humans. However, AIs are outperforming humans in many tasks. They are especially adept at taking standardized examinations, such as Board Exams, the Uniform Bar Exam, and the SAT and Graduate Record Exam. AIs are also performing diagnosis. Initial reports found that newer models have been observed to equal or outperform physicians in diagnostic accuracy and in the conveyance of empathy.In the area of interdiction, AI robots can both navigate and monitor for confrontational and illegal behavior. A human security agent would then be notified to resolve the situation. Our military is fielding autonomous AI robots to counter potential adversaries. For many, this new arms race has grave implications because of the potential of fielding this same security technology in healthcare and other civil settings.The healthcare delivery sector must determine the future roles of AI in relationship to human workers. AIs should only be used to support a human employee. AIs should not be the primary caregiver and a single human should not be monitoring multiple AIs simultaneously. Similar to not being copyrightable, disinformation produced by AIs should not be afforded 'free speech' protections. Any increase in productivity of an AI will equate with a loss of jobs. We need to ask, If all business sectors utilize AIs, will there be enough paid workers for the purchasing of services and products to keep our economy and society a float?

Examining the trajectory of relational conflict leading up to and after an anticipated stressor

AbstractA wealth of research suggests that as stress increases, so does conflict in close relationships. But is this always the case? Specifically, how does the trajectory of conflict unfold in the period before and after an acute and anticipated major life stressor? We capitalize on a 44‐day dyadic diary where one partner was preparing for the New York State Bar Exam (N = 312 couples) to explore the trajectory of conflict leading up to the exam, as well as how these patterns differed once exams finished. Results revealed that examinees and partners reported statistically meaningful decreases in conflict as exams approached, which contrasts prior research. This was followed by a spike in the likelihood of conflict immediately after exams were finished, and a quick return to low likelihoods of conflict one week after exams. This work highlights how couples, even in the face of stress, regulate their emotions and engage in relationship maintenance processes. However, relationship frustrations tend to be expressed—resulting in conflict—once the stressor is over. Not only does this work inform theories about relational conflict during times of stress, but it also highlights the need to study the dynamics of stress before, during, and after meaningful life events.

Does the 1L curriculum make a difference?

AbstractGeorgetown Law's Curriculum B (also known as Section 3) offers a unique opportunity to study an alternative 1L curriculum. The standard 1L curriculum has been around for decades and is still offered at the vast majority of US law schools. Leaders in the legal academy often talk about experimenting with the 1L curriculum, but hardly anyone does it. Georgetown Law has. We study whether Georgetown's Curriculum B yields measurable differences in student outcomes. Our empirical design leverages the fact that enrollment in Curriculum B is done by lottery when it is oversubscribed—meaning our study is effectively a randomized controlled trial. We measure treatment effects of Curriculum B by comparing outcomes of students who received the treatment (Curriculum B) with outcomes of students who received the placebo (Curriculum A) but wanted the treatment. Because students in both the treatment and control groups elected to enroll in Curriculum B, our empirical design overcomes the issue of selection bias. We find that taking Curriculum B decreases students' performance in two business law electives (Corporations and Securities Regulation) and reduces the rate at which they graduate with Latin honors. In addition, we find that it increases students' propensity to take certain public law electives and decreases their propensity to take certain business law electives. We further find that taking Curriculum B decreases students' likelihood of working in the private sector (law firm or business/industry), increases their likelihood of working in the public sector (government or public interest) or doing a judicial clerkship, and reduces their average annual salary. At the same, however, we find no statistically significant effects on other outcomes, including students' cumulative grade point average, their chances of passing the bar exam or being employed 10 months after graduation, or their rate or amount of alumni giving.

Re-evaluating GPT-4’s bar exam performance

AbstractPerhaps the most widely touted of GPT-4’s at-launch, zero-shot capabilities has been its reported 90th-percentile performance on the Uniform Bar Exam. This paper begins by investigating the methodological challenges in documenting and verifying the 90th-percentile claim, presenting four sets of findings that indicate that OpenAI’s estimates of GPT-4’s UBE percentile are overinflated. First, although GPT-4’s UBE score nears the 90th percentile when examining approximate conversions from February administrations of the Illinois Bar Exam, these estimates are heavily skewed towards repeat test-takers who failed the July administration and score significantly lower than the general test-taking population. Second, data from a recent July administration of the same exam suggests GPT-4’s overall UBE percentile was below the 69th percentile, and $$\sim$$ ∼ 48th percentile on essays. Third, examining official NCBE data and using several conservative statistical assumptions, GPT-4’s performance against first-time test takers is estimated to be $$\sim$$ ∼ 62nd percentile, including $$\sim$$ ∼ 42nd percentile on essays. Fourth, when examining only those who passed the exam (i.e. licensed or license-pending attorneys), GPT-4’s performance is estimated to drop to $$\sim$$ ∼ 48th percentile overall, and $$\sim$$ ∼ 15th percentile on essays. In addition to investigating the validity of the percentile claim, the paper also investigates the validity of GPT-4’s reported scaled UBE score of 298. The paper successfully replicates the MBE score, but highlights several methodological issues in the grading of the MPT + MEE components of the exam, which call into question the validity of the reported essay score. Finally, the paper investigates the effect of different hyperparameter combinations on GPT-4’s MBE performance, finding no significant effect of adjusting temperature settings, and a significant effect of few-shot chain-of-thought prompting over basic zero-shot prompting. Taken together, these findings carry timely insights for the desirability and feasibility of outsourcing legally relevant tasks to AI models, as well as for the importance for AI developers to implement rigorous and transparent capabilities evaluations to help secure safe and trustworthy AI.

Democratizing Law Librarianship: Reducing Barriers to Entry through Alternative Pathways to the Profession and Increased Support to Students. A Call to Action

Law librarianship is a constantly evolving profession driven by the evolution of law practice, legal education, government, and law itself. Changes in these drivers are in turn influenced by factors such as technology, culture, client needs, American Bar Association Standards, bar exams, diversity and access efforts, faculty research, instructional trends, and law school rankings. Law librarians proudly keep up with these changes—and even stay ahead of them—as we impart new knowledge and skills to users of law libraries and legal information resources. As we proceed through the third decade of the twenty-first century, the legal information profession is engaged in dialogue about the perpetually shrinking pools of qualified candidates for law librarian positions. Additionally, law librarians have been lamenting for decades that the legal information profession does not accurately reflect the diversity in our communities. The literature reflects that those conversations began in earnest in the 1970s and continue today. This article addresses both compelling issues and offers concrete strategies to tackle them simultaneously, thoughtfully, and intentionally. The entire profession is invited to play a role in this effort.

AI and access to justice: How AI legal advisors can reduce economic and shame-based barriers to justice

ChatGPT – a large language model – recently passed the U.S. bar exam. The startling rise and power of generative artificial intelligence (AI) systems such as ChatGPT lead us to consider whether and how more specialized systems could be used to overcome existing barriers to the legal system. Such systems could be employed in either of the two major stages of the pursuit of justice: preliminary information gathering and formal engagement with the state’s legal institutions and professionals. We focus on the former and argue that developing and deploying publicly funded AI legal advisors can reduce economic and shame-based cultural barriers to the information-gathering stage of pursuing justice.

Contribution Analysis of Large Language Models and Data Augmentations for Person Names in Solving Legal Bar Examination at COLIEE 2023

This paper describes our system for COLIEE 2023 Task 4, which automatically answers Japanese legal bar exam problems. We propose an extension to our previous system in COLIEE 2022, which achieved the highest accuracy among all submissions using data augmentation. We focus on problems that include mentions of person names. In this paper, we present two main contributions. First, we incorporate LUKE as our deep learning component, which is a named entity recognition model trained on RoBERTa. Second, we fine-tune the pretrained LUKE model in multiple ways, comparing fine-tuning on training datasets that include alphabetical person names and ensembling different fine-tuning models. We confirmed that LUKE and its fine-tuned model on person type problems improve their accuracies. Our formal run results show that LUKE and our fine-tuning approach using alphabetical person names were effective, achieving an accuracy of 0.69 in the COLIEE 2023 Task 4 formal run.

In Case You Haven't Heard…

Being in treatment on buprenorphine or methadone can get you in trouble. Yet another patient in recovery from opioid use disorder has had his treatment used against him. A young man who proudly passed the bar exam and had an employment contract with a law firm was told by Tennessee state authorities that they doubted his ability to practice law because he was taking buprenorphine, a medication that had helped him achieve what he called “normalcy.” An excellent write‐up aired by NBC News last week described the ordeal of Derek Scott, the first in his family to even go to high school, and then later, after struggling with opioid addiction and other problems, to have achieved recovery and enrolled in college at age 32. Sally Friedman of the Legal Action Center discussed the many violations of the Americans with Disabilities Act this case entails, with legal officials, of all people, in charge. The U.S. Department of Justice has been investigating such cases and is investigating Scott's. Stay tuned. For the story, go to https://www.nbcnews.com/news/us‐news/tennessee‐lawyer‐opioid‐addiction‐medication‐ada‐discrimination‐rcna126358.

Recognizing Textual Inference in Mongolian Bar Exam Questions

This paper examines how to apply deep learning techniques to Mongolian bar exam questions. Several approaches that utilize eight different fine-tuned transformer models were demonstrated for recognizing textual inference in Mongolian bar exam questions. Among eight different models, the fine-tuned bert-base-multilingual-cased obtained the best accuracy of 0.7619. The fine-tuned bert-base-multilingual-cased was capable of recognizing “contradiction”, with a recall of 0.7857 and an F1 score of 0.7674; it recognized “entailment” with a precision of 0.7750, a recall of 0.7381, and an F1 score of 0.7561. Moreover, the fine-tuned bert-large-mongolian-uncased showed balanced performance in recognizing textual inference in Mongolian bar exam questions, thus achieving a precision of 0.7561, a recall of 0.7381, and an F1 score of 0.7470 for recognizing “contradiction”.

Estimating the Returns to Occupational Licensing: Evidence from Regression Discontinuities at the Bar Exam

Estimating the Returns to Occupational Licensing: Evidence from Regression Discontinuities at the Bar Exam

Are Law Schools Cream-Skimming to Bolster Their Bar Exam Pass Rates? A Multilevel Regression Approach to Estimate How Attrition and Transfer Rates Affect Bar Passage

Law schools are held accountable on many fronts to achieve and maintain high bar passage rates. While the course of legal education itself, along with various interventions, is a key driver of bar exam performance, Bahadur et al. (2021) suggests that obscure institutional practices might be inflating institutional bar passage performance. Such practices could include recruitment and admission of transfer students and academic attrition. We examine this hypothesis to assess the influence of both attrition and transfer on law schools’ bar passage differential using fixed-effects and between-within models. We also utilize Poisson regression to explore the effect of geographical proximity to other law schools on transfer rates. We find that, on average, neither attrition nor transfer activity substantively affect bar passage performance, and that although geographic proximity to other schools is related to transfer rates, this relationship does not result in notable differences in bar passage.

Next-Gen Bar Exam That Truly Tests Daily Practice Skills Must Include Technology

Next-Gen Bar Exam That Truly Tests Daily Practice Skills Must Include Technology

Whither Coursework and NextGen Bar Exam Success?

Whither Coursework and NextGen Bar Exam Success?

Breaking the Cycle: Rethinking Bar Exam Scoring and Portability for NextGen Examinees. 

Breaking the Cycle: Rethinking Bar Exam Scoring and Portability for NextGen Examinees. 

Supreme Court (Plenary Session) Decisions 2022

The Thai Supreme Court, also referred to as San Dika, is the final adjudicator for civil and criminal cases in Thailand. Supreme Court decisions are informally authoritative in the Thai legal system, with practitioners using past decisions to support their arguments and students studying past decisions to pass the bar exam. As a deliberation of the Supreme Court in its plenary session requires the attendance of not less than half of the total number of judges, decisions with the involvement of plenary sessions are therefore deemed even more authoritative
 This compilation seeks to summarise, in English, 13 plenary session judgements of the Supreme Court in 2022. Through such a task, it is hoped that this will help encourage like-minded individuals to make Thai jurisprudence more internationally accessible.

Evaluating ChatGPT Performance on the Orthopaedic In-Training Examination.

Artificial intelligence (AI) holds potential in improving medical education and healthcare delivery. ChatGPT is a state-of-the-art natural language processing AI model which has shown impressive capabilities, scoring in the top percentiles on numerous standardized examinations, including the Uniform Bar Exam and Scholastic Aptitude Test. The goal of this study was to evaluate ChatGPT performance on the Orthopaedic In-Training Examination (OITE), an assessment of medical knowledge for orthopedic residents. OITE 2020, 2021, and 2022 questions without images were inputted into ChatGPT version 3.5 and version 4 (GPT-4) with zero prompting. The performance of ChatGPT was evaluated as a percentage of correct responses and compared with the national average of orthopedic surgery residents at each postgraduate year (PGY) level. ChatGPT was asked to provide a source for its answer, which was categorized as being a journal article, book, or website, and if the source could be verified. Impact factor for the journal cited was also recorded. ChatGPT answered 196 of 360 answers correctly (54.3%), corresponding to a PGY-1 level. ChatGPT cited a verifiable source in 47.2% of questions, with an average median journal impact factor of 5.4. GPT-4 answered 265 of 360 questions correctly (73.6%), corresponding to the average performance of a PGY-5 and exceeding the corresponding passing score for the American Board of Orthopaedic Surgery Part I Examination of 67%. GPT-4 cited a verifiable source in 87.9% of questions, with an average median journal impact factor of 5.2. ChatGPT performed above the average PGY-1 level and GPT-4 performed better than the average PGY-5 level, showing major improvement. Further investigation is needed to determine how successive versions of ChatGPT would perform and how to optimize this technology to improve medical education. AI has the potential to aid in medical education and healthcare delivery.

Reforging the Master’s Tools: Critical Race Theory in the First-Year Curriculum

This Article examines why and how critical race theory (CRT) should be taught as a mandatory component of the first-year law school curriculum. Learning the fundamentals of critical race theory is not only important to empathetically understand and serve those around you, but necessary to understand the law as it is. The law’s past and future require this. This Article first makes the positive argument for critical race theory’s necessity in legal education, showing that it rises above normative (albeit virtuous) justifications. It then briefly summarizes what critical race theory is by outlining its central tenets, as well as what critical race theory is not by examining the recent uproar surrounding the CRT boogeyman. Part II explores why and how critical race theory can be taught in two doctrinal first-year courses: constitutional law and civil procedure. The courses show how easily critical race theory’s tenets slot into commonly taught cases, before even considering adding new material to course requirements. Constitutional law demonstrates how interest convergence and the social construction of race play a role in the law, while civil procedure demonstrates how intersectionality and counter-storytelling can recontextualize how we view a court’s ruling. Part III prescribes actionable solutions. The first solutions include discretionary implementation of critical race theory into syllabi by professors. It suggests more concrete solutions in the form of mandatory curriculum requirements, changes to ABA accreditation standards, and changes in NCBE Bar Exam requirements. It then surveys some existing courses to examine how such requirements can be implemented. Finally, the appendices outline the wealth of existing CRT scholarship relating to the first-year courses for reference.

News Article

Possible impacts of Large Language Models on X-ray spectroscopy (March 17, 2023). Recent announcement of releasing OpenAI's Generative Pretrained Transformer 4 (GPT-4) could mark the beginning of a new era of the practical application of artificial intelligence. Compared with the previous versions of OpenAI's Large Language Models (LLM), its function in handling the images even in the text-based chat system is surprising. In the presentation, which has been archived as a Youtube video (https://www.youtube.com/live/outcGtbnMuQ?feature=share), the chat system can handle a screenshot image of a handwriting note with some sketch, and can instantly transform it into an html code. Then one can create a new Web page. In addition, the reasoning ability has been upgraded considerably from the previous version, which had not been very good; previous ChatGPT seems to give the correct answer only to 26% of the junior high school level mathematics problems (see, for example, Simon Frieder et al., “Mathematical Capabilities of ChatGPT”, https://arxiv.org/abs/2301.13867). The new version seems to be much better and is now quite good at the tax calculation and the bar exam problem. Generally, the ability of text-based job has become further better than ever. The number of characters that can be handled has increased, and overall processing has become higher quality. One of the recent interesting discussions on the application of artificial intelligence to spectroscopy has been described in the article, by Jerome Workman, Jr., and Howard Mark, “Artificial Intelligence in Analytical Spectroscopy, Part I: Basic Concepts and Discussion”, Spectroscopy, 38, (2), 13–22 (2023). https://doi.org/10.56530/spectroscopy.og4284z8. Some of the related contents have been also discussed in the podcast, Analytically Speaking Podcast Episode 9, “Automating Advanced Chemometric Methods for Data Processing” (https://www.spectroscopyonline.com/view/ep-9-automating-advanced-chemometric-methods-for-data-processing). As the present artificial intelligence is more powerful in handling billions of images, for a while, image-based scientific applications could be preceding (see, for example, Lei Zhang and Shaofeng Shao, “Image-based machine learning for materials science”, Journal of Applied Physics 132, 100701 (2022). https://doi.org/10.1063/5.0087381; Samantha Phan and Christine K. Luscombe, “Recent trends in marine microplastic modeling and machine learning tools: Potential for long-term microplastic monitoring”, Journal of Applied Physics 133, 020701 (2023). https://doi.org/10.1063/5.0126358). On the same day of the announcement of GPT-4, Anthropic Inc, which is a rival company founded by former Open AI employees, released Claude. It has been tested in private systems on systems such as Notion, Quora, and DuckDuckGo, and has been very well received. The LLM and its application such as chatbots will be used more frequently than before. On the other hand, the system is like a black box, and therefore we will need to be careful when considering some real use (see, for example, Noah Giansiracusa, “Three Easy Ways to Make AI Chatbots Safer”, Scientific American, March 17, 2023, https://www.scientificamerican.com/article/three-easy-ways-to-make-ai-chatbots-safer/).