Keynote - AI for the Public Sector and the Case of Legal NLP

Abstract

Recent innovations such as ChatGPT have increased public interest in artificial intelligence (AI). The keynote explained why AI is not just a short-term hype but has a long history of spanning several eras. A recent revolution has been in the field of Natural Language Processing (NLP). This interdisciplinary field of research is also known as computational linguistics. It is usually implemented by specific NLP tasks, ranging from simple processing steps such as tokenization, stemming, lemmatization to Part of Speech (PoS) tagging and topic modeling. A second, more complex set of NLP tasks includes Namend Entity Recognition (NER), information retrieval, relationship extraction, sentiment analysis, text similarity, and coreference resolution. Finally, the most challenging NLP tasks are considered Question Answering (QA), text summarization, text simplification, text generation, text translation, and chatbots. NLP has especially great potential in the public sector. For example, a new multilingual legal language model for more than 20 languages, developed for the Swiss Federal Court, offers opportunities to increase accessibility of legal documents for citizens while preserving the digital sovereignty of government institutions. These technical results of the National Research Program (NRP) 77 project “Open Justice versus Privacy” are published on Hugging Face, a platform for sharing openly available machine learning models and datasets. Today, it is mostly private companies that build such Large Language Models (LLM), because it requires a large amount of computational resources and highly skilled engineers. For example, to train the new LLaMA model, Meta AI (Facebook) needed more than $30 million worth of graphical processing units (GPU). In addition, 450 MWh of electricity worth about $90,000 was needed to process the data on these GPUs. Negative for innovation and the environment, Meta AI released the LLaMA model only under a non-commercial license. This means that startups and other companies cannot use the model for their own services. This calls for a discussion about how “open” today's machine learning models should be and what “open” actually means in the age of AI. The keynote presentation therefore included a proposal of 5 elements of such machine learning models that need to be openly available and licensed under an official open license in order to speak of an Open AI Model. This term is used by the United Nations definition of Digital Public Goods. These five elements include 1) model architecture (detailed scientific publications), 2) hyperparameters (built configuration), 3) training data (labeled and unlabeled datasets), 4) model weights and intermediate checkpoints (parameters), and 5) source code to build the model (programming scripts etc.). A truly openly available AI model is BLOOM, an LLM from the BigScience initiative. It was built by more than 1000 researchers from over 70 countries, trained on an infrastructure that would have cost EUR 3 million. BLOOM was released on July 12th, 2022 on Hugging Face and is licensed under the Responsible AI License (RAIL), a new type of AI license that incorporates ethical aspects while preserving the openness of the machine learning elements described.