Domain-specific Modeling Languages Research Articles

Machine learning models for abstract screening task - A systematic literature review application for health economics and outcome research

ObjectiveSystematic literature reviews (SLRs) are critical for life-science research. However, the manual selection and retrieval of relevant publications can be a time-consuming process. This study aims to (1) develop two disease-specific annotated corpora, one for human papillomavirus (HPV) associated diseases and the other for pneumococcal-associated pediatric diseases (PAPD), and (2) optimize machine- and deep-learning models to facilitate automation of the SLR abstract screening.MethodsThis study constructed two disease-specific SLR screening corpora for HPV and PAPD, which contained citation metadata and corresponding abstracts. Performance was evaluated using precision, recall, accuracy, and F1-score of multiple combinations of machine- and deep-learning algorithms and features such as keywords and MeSH terms.Results and conclusionsThe HPV corpus contained 1697 entries, with 538 relevant and 1159 irrelevant articles. The PAPD corpus included 2865 entries, with 711 relevant and 2154 irrelevant articles. Adding additional features beyond title and abstract improved the performance (measured in Accuracy) of machine learning models by 3% for HPV corpus and 2% for PAPD corpus. Transformer-based deep learning models that consistently outperformed conventional machine learning algorithms, highlighting the strength of domain-specific pre-trained language models for SLR abstract screening. This study provides a foundation for the development of more intelligent SLR systems.

Do domain-specific protein language models outperform general models on immunology-related tasks?

Do domain-specific protein language models outperform general models on immunology-related tasks?

인공지능기반 사전학습언어모델 적용방안에 관한 연구

Pre-trained Language Model(PLM) refers to a natural language processing(NLP) model that has been pre-trained using large amounts of text data. The PLM has the limitation of not being able to understand domain-specific terminology due to a lack of training data for terminology. Therefore, the need for a domain-specific language model modified through BERT- or GPT-based pre-trained learning has recently been emphasized. In this study, we analyze BERT's pre-training method and BERT-based transformation techniques (ALBERT, RoBERTa, ELECTRA) and propose a PLM that can be used in biomedical, financial, and legal domains. The biomedical-specific pre-trained learning model is designed to learn domain-specific language characteristics such as technical terminology, medical sentence structure, and medical entity name recognition in the biomedical field. It is mainly adjusted to be applied to biomedical tasks through transfer learning based on BERT's pre-training method and architecture. For this purpose, it is pre-trained with pre-trained biomedical text data, and this pre-training transfers domain-specific knowledge to the model through learning representations for biomedical-related texts. The finance-specific pre-trained learning model is a model that can understand and process financial terminology, financial market trends, and sentence structures and vocabulary related to financial products and services. It can be used to generate news articles about financial market trends and to extract key information by concisely summarizing long texts such as financial reports and corporate press releases. Additionally, finance-specific pre-trained models help financial analysts generate investment recommendations based on a company's financial condition, performance, and prospects. The legal-specific pre-trained model is a language model suitable for legal documents and is used for legal document classification, legal document summarization, and legal document similarity evaluation. The legal-specific pre-learning model was created by pre-training the BERT model on special texts in the legal field, and through this, it learns characteristics specialized for legal documents. The performance of the legal-specific pre-training model can be improved to solve legal-related tasks through scratch pre-training and additional pre-training using legal corpora.

Effective Natural Language Processing Algorithms for Early Alerts of Gout Flares from Chief Complaints

Early identification of acute gout is crucial, enabling healthcare professionals to implement targeted interventions for rapid pain relief and preventing disease progression, ensuring improved long-term joint function. In this study, we comprehensively explored the potential early detection of gout flares (GFs) based on nurses’ chief complaint notes in the Emergency Department (ED). Addressing the challenge of identifying GFs prospectively during an ED visit, where documentation is typically minimal, our research focused on employing alternative Natural Language Processing (NLP) techniques to enhance detection accuracy. We investigated GF detection algorithms using both sparse representations by traditional NLP methods and dense encodings by medical domain-specific Large Language Models (LLMs), distinguishing between generative and discriminative models. Three methods were used to alleviate the issue of severe data imbalances, including oversampling, class weights, and focal loss. Extensive empirical studies were performed on the Gout Emergency Department Chief Complaint Corpora. Sparse text representations like tf-idf proved to produce strong performances, achieving F1 scores higher than 0.75. The best deep learning models were RoBERTa-large-PM-M3-Voc and BioGPT, which had the best F1 scores for each dataset, with a 0.8 on the 2019 dataset and a 0.85 F1 score on the 2020 dataset, respectively. We concluded that although discriminative LLMs performed better for this classification task when compared to generative LLMs, a combination of using generative models as feature extractors and employing a support vector machine for classification yielded promising results comparable to those obtained with discriminative models.

Development of a Language Model for Named-Entity-Recognition in Aerospace Requirements

We address the challenges inherent in converting natural language (NL) requirements into machine-readable formats by investigating the application of named-entity recognition (NER) within the aerospace domain. Recognizing the necessity for domain-specific language models, we developed an open-source annotated aerospace corpus and fine-tuned different versions of the BERT language model on the corpus to create aeroBERT-NER: a new model for identifying named entities (NEs) in the aerospace domain. A comparison between aeroBERT-NER and BERTBASE-NER demonstrated the superior performance of aeroBERT-NER in identifying NEs within a set of aerospace requirements. The identified NEs contribute to the development of a glossary, promoting consistent terminology usage in aerospace requirements and addressing challenges associated with the standardization of NL requirements.

Empirical evaluation of a threat modeling language as a cybersecurity assessment tool

The complexity of ICT infrastructures is continuously increasing, presenting a formidable challenge in safeguarding them against cyber attacks. In light of escalating cyber threats and limited availability of expert resources, organizations must explore more efficient approaches to assess their resilience and undertake proactive measures. Threat modeling is an effective approach for assessing the cyber resilience of ICT systems. One method is to utilize Attack Graphs, which visually represent the steps taken by adversaries during an attack. Previously, MAL (the Meta Attack Language) was proposed, which serves as a framework for developing Domain-Specific Languages (DSLs) and generating Attack Graphs for modeled infrastructures. coreLang is a MAL-based threat modeling language that utilizes such Attack Graphs to enable attack simulations and security assessments for the generic ICT domain. Developing domain-specific languages for threat modeling and attack simulations provides a powerful approach for conducting security assessments of infrastructures. However, ensuring the correctness of these modeling languages raises a separate research question. In this study we conduct an empirical experiment aiming to falsify such a domain-specific threat modeling language. The potential inability to falsify the language through our empirical testing would lead to its corroboration, strengthening our belief in its validity within the parameters of our study. The outcomes of this approach indicated that, on average, the assessments generated by attack simulations outperformed those of human experts. Additionally, both human experts and simulations exhibited significantly superior performance compared to random guessers in their assessments. While specific human experts occasionally achieved better assessments for particular questions in the experiments, the efficiency of simulation-generated assessments surpasses that of human domain experts.

NanoBERT: a deep learning model for gene agnostic navigation of the nanobody mutational space.

Nanobodies are a subclass of immunoglobulins, whose binding site consists of only one peptide chain, bestowing favorable biophysical properties. Recently, the first nanobody therapy was approved, paving the way for further clinical applications of this antibody format. Further development of nanobody-based therapeutics could be streamlined by computational methods. One of such methods is infilling-positional prediction of biologically feasible mutations in nanobodies. Being able to identify possible positional substitutions based on sequence context, facilitates functional design of such molecules. Here we present nanoBERT, a nanobody-specific transformer to predict amino acids in a given position in a query sequence. We demonstrate the need to develop such machine-learning based protocol as opposed to gene-specific positional statistics since appropriate genetic reference is not available. We benchmark nanoBERT with respect to human-based language models and ESM-2, demonstrating the benefit for domain-specific language models. We also demonstrate the benefit of employing nanobody-specific predictions for fine-tuning on experimentally measured thermostability dataset. We hope that nanoBERT will help engineers in a range of predictive tasks for designing therapeutic nanobodies. https://huggingface.co/NaturalAntibody/.

BactInt: A domain driven transfer learning approach for extracting inter-bacterial associations from biomedical text

BackgroundThe healthy as well as dysbiotic state of an ecosystem like human body is known to be influenced not only by the presence of the bacterial groups in it, but also with respect to the associations within themselves. Evidence reported in biomedical text serves as a reliable source for identifying and ascertaining such inter bacterial associations. However, the complexity of the reported text as well as the ever-increasing volume of information necessitates development of methods for automated and accurate extraction of such knowledge. MethodsA BioBERT (biomedical domain specific language model) based information extraction model for bacterial associations is presented that utilizes learning patterns from other publicly available datasets. Additionally, a specialized sentence corpus has been developed to significantly improve the prediction accuracy of the ‘transfer learned’ model using a fine-tuning approach. ResultsThe final model was seen to outperform all other variations (non-transfer learned and non-fine-tuned models) as well as models trained on BioGPT (a domain trained Generative Pre-trained Transformer). To further demonstrate the utility, a case study was performed using bacterial association network data obtained from experimental studies. ConclusionThis study attempts to demonstrate the applicability of transfer learning in a niche field of life sciences where understanding of inter bacterial relationships is crucial to obtain meaningful insights in comprehending microbial community structures across different ecosystems. The study further discusses how such a model can be further improved by fine tuning using limited training data. The results presented and the datasets made available are expected to be a valuable addition in the field of medical informatics and bioinformatics.

SOAP classifier for free-text clinical notes with domain-specific pre-trained language models

The increasing use of electronic health records (EHRs) in healthcare has led to a significant amount of unstructured clinical text data. This paper proposes a model for classifying free-text clinical notes in the Portuguese language in sentences following the SOAP (Subjective, Objective, Assessment, and Plan) note standard using domain-specific pre-trained language models. Among the five pre-trained BERT models tested, BioBERTptRT achieved the best results with a precision of 0.9461, accuracy of 0.9434, recall of 0.9437, and F1-score of 0.9435. BioBERTptRT, specialized in the Portuguese language, clinical terminology, and the medical group’s domain, outperformed the other models, showing a 0.28% increase in the F1-score compared to the second most specialized model. The proposed model focuses on high-level sentence classification rather than entity-level classification and aims to structure clinical notes at a broader level. The study utilizes a private database of 10,000 anonymized health records containing 234,673 clinical notes. These notes were divided into 1,183,345 unique sentences used to retrain BioBERTptRT. Additionally, 100,021 sentences were manually labeled for use in fine-tuning the models This work contributes to the structuring of clinical notes by showcasing the performance improvements obtained through domain specialization in BERT networks. Additionally, it presents an analysis of the performance gains achieved by BioBERTpt compared to mBERT, both in our study and other related works. Furthermore, this study provides a valuable comparison between the distribution of sentences and the results obtained in this research and similar studies conducted in English.

An executable domain-specific modeling language for simulating organizational auction-based coordination strategies for crisis response

Emergency response environments in which the crisis occurs include many homogenous and heterogeneous entities, e.g., organizations and persons. These entities need to collaborate to address a crisis. However, most of the time, the organizations that are involved in a crisis suffer from communication and coordination problems. This may cause unfair resource distribution. To mitigate the problem, there is a pressing need to construct models of emergency response environments and simulate the coordination strategies proactively, before the onset of any crisis. This paper highlights the complexity of the system formed by interacting organizations during a crisis and the difficulty in programming for the simulation of such complex systems. To cope with this issue, this paper presents an executable domain-specific modeling language called CoorERE. The language facilitates modeling emergency response environments for the simulation of one of the most common coordination strategies, i.e., auction-based coordination strategies in crisis response. CoorERE is used for the graphical modeling of emergency response environments and simulation of both inter and intra-organizational auction-based coordination strategies. The paper presents two case studies with different complexities to show the applicability of CoorERE. An empirical study is conducted to compare CoorERE with MDD4ABMS, which is one of the most recent tools for modeling and simulation of multi-agent systems. The evaluation includes both objective and subjective assessments. The objective assessment considers the development effort required for modeling a complete case study, while the subjective assessment compares the tools in terms of satisfying modeling method requirements and quality metrics such as usability, productivity, and expressiveness. The results of the evaluation indicate that CoorERE reduces the development effort by about 47%. In terms of quality metrics, CoorERE obtained higher scores than MDD4ABMS for all the measurements considered, indicating its superior usability, productivity, and expressiveness.

Improving Medical Entity Recognition in Spanish by Means of Biomedical Language Models

Named Entity Recognition (NER) is an important task used to extract relevant information from biomedical texts. Recently, pre-trained language models have made great progress in this task, particularly in English language. However, the performance of pre-trained models in the Spanish biomedical domain has not been evaluated in an experimentation framework designed specifically for the task. We present an approach for named entity recognition in Spanish medical texts that makes use of pre-trained models from the Spanish biomedical domain. We also use data augmentation techniques to improve the identification of less frequent entities in the dataset. The domain-specific models have improved the recognition of name entities in the domain, beating all the systems that were evaluated in the eHealth-KD challenge 2021. Language models from the biomedical domain seem to be more effective in characterizing the specific terminology involved in this task of named entity recognition, where most entities correspond to the "concept" type involving a great number of medical concepts. Regarding data augmentation, only back translation has slightly improved the results. Clearly, the most frequent types of entities in the dataset are better identified. Although the domain-specific language models have outperformed most of the other models, the multilingual generalist model mBERT obtained competitive results.

Fine-grained adaptive parallelism for automotive systems through AMALTHEA and OpenMP

The software development complexity of automotive systems has significantly increased during the last decade due to the latest Advanced Driving Assistance System (ADAS) functionalities. To effectively address this complexity, domain specific modeling languages (DSMLs) like AUTOSAR or an open-source system performance model for AUTOSAR-aligned systems, APP4MC, have become a common trend in the automotive industry. DSMLs allow for easily capturing the functional and non-functional requirements of the system without needing to master low level details of the programming model or the processor architecture. Unfortunately, current DSMLs do not support the parallel programming models, like OpenMP and CUDA, that are used to exploit parallel heterogeneous architectures featuring acceleration devices such as GPUs and FPGAs required. These architectures are however essential to cope with the performance needs of ADAS. This exposes a gap between the DSMLs used by automotive designers to enhance software productivity and leverage verification and validation processes, and the parallel processor architectures used in this domain. This paper presents a complete framework to safely exploit the inherent parallelism exposed by the AMALTHEA system description, supported in APP4MC, by: (1) automatically transforming the high-level design into the OpenMP parallel programming model targeting both host and accelerator parallelism, and (2) using compiler analysis techniques to prove the correctness of the model transformed to OpenMP code. The paper contributes also with (3) an analysis of the parallel execution model allowed by the AMALTHEA DSML and that of OpenMP, and (4) a performance plus productivity evaluation of the proposed framework on real automotive systems executed on an embedded GPU-based processor architecture.

Synthetic Histopathological Images Generation with Artificial Intelligence to Accelerate Research and Improve Clinical Outcomes in Hematology

Background: Hematological malignancies are rare and complex diseases and as a consequence, multimodal data (ranging from clinical and genomic information to images) are required to improve diagnosis, prognosis and personalized treatments. However, collecting all these layers of information is challenging, in particular when collecting cytological and histological images from the bone marrow (BM) reproducing disease morphologic features. Synthetic data generation by Artificial Intelligence (AI) can circumvent these issues by generating images conditioned from textual inputs (i.e. reports from pathologists), which are widely available and contain many useful clinical information. This technology can enrich data with synthetic images, thus boosting translational research and improving the performances of precision medicine strategies based on multimodal information. Aims:This project was conducted by GenoMed4all and Synthema EU consortia, with the aim to:1)Apply generative models to real-world dataset with histological images of patients with myeloid neoplasms (MN). 2) Develop a Synthetic Images Validation Framework (SIVF) to evaluate the utility and fidelity of generated images. 3) Verify the capability of synthetic images to accelerate research and to improve clinical models. Methods:We implemented Stable Diffusion (SD) generative model fine-tuned on hematological data to generate Hematoxylin and Eosin (H&E) images of MN patients. We implemented a domain specific language model (HematoBERT) to encode textual input as condition for the generation process. Use cases were Myelodysplastic Syndrome (MDS), Acute Myeloid Leukemia (AML) and Myeloproliferative Neoplasm (MPN) patients, with available BM biopsies and their reports from pathologists, genomic and clinical data. We applied SIVF to evaluate distributions of morphological features extracted from real and synthetic images. Clinical validation was performed on disease classification and survival probability prediction, using real and synthetic images features (experimental setting is reported in Figure 1). Results: We trained SD model on 200 patients with available BM biopsies and associated reports. We first performed SIVF to compare extracted morphological features (geometrical, color and texture features of cells nuclei) from synthetic and real images of 55 patients never seen by the model. Results proved that features distributions and correlations in both datasets were comparable. Similar results were obtained performing SIVF on each single patient data. We verified if synthetic data augmentation could improve performances on MN classification (i.e. models able to correctly assign a single patient to a specific clinical entity according to the 2022 WHO classification criteria). We implemented three XGBOOST models to classify patients' disease. Classifiers were trained and validated on morphological extracted features of images from a real set of patients (n=614), a synthetic group (n=396) and a mixed dataset (n=1010). Data augmentation improved classification performance by 10% (F1 Score) when we tested it on the three different validation sets. Finally, demographics, clinical features, genomics (cytogenetics and gene mutations) were included as covariates together with morphological features extracted from BM biopsies in L1 penalized Cox's proportional hazards models, considering Overall Survival as primary endpoint. Models were fitted on two different cohorts of real patients (n=182, n=294). Then we added 112 synthetic patients to both sets and refitted the models. We observed an improvement in performances of >10% (C-Index) for both cases (Figure 2), with morphological features (such as “major axis” of nuclei) being selected among the best predictors. All these results confirmed that data augmentation through synthetic data is a viable approach and can significantly improve the models capability to efficiently capture clinical outcomes at individual patient level. Conclusion:AI generated images preserve properties of real-world images, replicating cells morphological features relevant to identify hematological diseases and their clinical status. This approach based on widely available textual data allows effective data augmentation and effortless data sharing, thus accelerating and improving precision medicine research in hematology.

Toplu Taşıma İçin Nesnelerin İnterneti Yazılımlarının Modellenmesi

The Internet of Things (IoT) is a very promising domain and it is one of the leading technologies used in the public transportation in recent years. However, in addition to the heterogeneity and high complexity problems which are usually observed in the development of IoT systems, the specific needs of public transportation domain make the construction of such systems even harder for the public transportation. This paper proposes the use of a domain-specific modelling language (DSML), called DSML4PT, to facilitate the design and implementation of IoT-based public transportation systems. A metamodel is introduced that enables modeling IoT-based applications according to the different viewpoints and leads to the model-driven engineering of such applications for different IoT-based public transportation platforms. Furthermore, originated from this metamodel, design and implementation of the DSML4PT language with including its syntax and semantics definitions are all discussed in this paper. Use of this DSML supports both the design of the IoT-based public transportation software graphically and the automatic generation of the code required for the implementation. Based on the conducted case study, it has been observed that 80% of a public transportation application can be generated only with using DSML4PT.

A text classification-based approach for evaluating and enhancing the machine interpretability of building codes

Interpreting regulatory documents or building codes into computer-processable formats is essential for the intelligent design and construction of buildings and infrastructures. Although automated rule interpretation (ARI) methods have been investigated for years, most of them are highly dependent on the early and manual filtering of interpretable clauses from a building code. While few of them considered machine interpretability, which represents the potential to be transformed into a computer-processable format, from both clause- and document-level. Therefore, this research aims to propose a novel approach to automatically evaluate and enhance the machine interpretability of single clauses and building codes. First, a few categories are introduced to classify each clause in a building code considering the requirements for rule interpretation, and a dataset is developed for model training. Then, an efficient text classification model is developed based on a pretrained domain-specific language model and transfer learning techniques. Finally, a quantitative evaluation method is proposed to assess the overall interpretability of building codes. Experiments show that the proposed text classification algorithm outperforms the existing CNN- or RNN-based methods, by improving the F1-score from 72.16% to 93.60%. It is also illustrated that the proposed classification method can enhance downstream ARI methods with an improvement of 4%. Furthermore, analysis of more than 150 building codes in China showed that their average interpretability is only 34.40%, which implies that it is still difficult to fully transform an entire regulatory document into computer-processable formats. It is also argued that the interpretability of building codes should be further improved both from the human side (considering certain constraints when writing building codes) and the machine side (developing more powerful algorithms, tools, etc.).

A self-supervised language model selection strategy for biomedical question answering

Large neural-based Pre-trained Language Models (PLM) have recently gained much attention due to their noteworthy performance in many downstream Information Retrieval (IR) and Natural Language Processing (NLP) tasks. PLMs can be categorized as either general-purpose, which are trained on resources such as large-scale Web corpora, and domain-specific which are trained on in-domain or mixed-domain corpora. While domain-specific PLMs have shown promising performance on domain-specific tasks, they are significantly more computationally expensive compared to general-purpose PLMs as they have to be either retrained or trained from scratch. The objective of our work in this paper is to explore whether it would be possible to leverage general-purpose PLMs to show competitive performance to domain-specific PLMs without the need for expensive retraining of the PLMs for domain-specific tasks. By focusing specifically on the recent BioASQ Biomedical Question Answering task, we show how different general-purpose PLMs show synergistic behaviour in terms of performance, which can lead to overall notable performance improvement when used in tandem with each other. More concretely, given a set of general-purpose PLMs, we propose a self-supervised method for training a classifier that systematically selects the PLM that is most likely to answer the question correctly on a per-input basis. We show that through such a selection strategy, the performance of general-purpose PLMs can become competitive with domain-specific PLMs while remaining computationally light since there is no need to retrain the large language model itself. We run experiments on the BioASQ dataset, which is a large-scale biomedical question-answering benchmark. We show that utilizing our proposed selection strategy can show statistically significant performance improvements on general-purpose language models with an average of 16.7% when using only lighter models such as DistilBERT and DistilRoBERTa, as well as 14.2% improvement when using relatively larger models such as BERT and RoBERTa and so, their performance become competitive with domain-specific large language models such as PubMedBERT.

Forecasting the S&P 500 Index Using Mathematical-Based Sentiment Analysis and Deep Learning Models: A FinBERT Transformer Model and LSTM

Stock price prediction has been a subject of significant interest in the financial mathematics field. Recently, interest in natural language processing models has increased, and among them, transformer models, such as BERT and FinBERT, are attracting attention. This study uses a mathematical framework to investigate the effects of human sentiment on stock movements, especially in text data. In particular, FinBERT, a domain-specific language model based on BERT tailored for financial language, was employed for the sentiment analysis on the financial texts to extract sentiment information. In this study, we use “summary” text data extracted from The New York Times, representing concise summaries of news articles. Accordingly, we apply FinBERT to the summary text data to calculate sentiment scores. In addition, we employ the LSTM (Long short-term memory) methodology, one of the machine learning models, for stock price prediction using sentiment scores. Furthermore, the LSTM model was trained by stock price data and the estimated sentiment scores. We compared the predictive power of LSTM models with and without sentiment analysis based on error measures such as MSE, RMSE, and MAE. The empirical results demonstrated that including sentiment scores through the LSTM model led to improved prediction accuracy for all three measures. These findings indicate the significance of incorporating news sentiment into stock price predictions, shedding light on the potential impact of psychological factors on financial markets. By using the FinBERT transformer model, this study aimed to investigate the interplay between sentiment and stock price predictions, contributing to a deeper understanding of mathematical-based sentiment analysis in finance and its role in enhancing forecasting in financial mathematics. Furthermore, we show that using summary data instead of entire news articles is a useful strategy for mathematical-based sentiment analysis.

SWEL: A Domain-Specific Language for Modeling Data-Intensive Workflows

AbstractData-intensive applications aim at discovering valuable knowledge from large amounts of data coming from real-world sources. Typically, workflow languages are used to specify these applications, and their associated engines enable the execution of the specifications. However, as these applications become commonplace, new challenges arise. Existing workflow languages are normally platform-specific, which severely hinders their interoperability with other languages and execution engines. This also limits their reusability outside the platforms for which they were originally defined. Following the Design Science Research methodology, the paper presents SWEL (Scientific Workflow Execution Language). SWEL is a domain-specific modeling language for the specification of data-intensive workflows that follow the model-driven engineering principles, covering the high-level definition of tasks, information sources, platform requirements, and mappings to the target technologies. SWEL is platform-independent, enables collaboration among data scientists across multiple domains and facilitates interoperability. The evaluation results show that SWEL is suitable enough to represent the concepts and mechanisms of commonly used data-intensive workflows. Moreover, SWEL facilitates the development of related technologies such as editors, tools for exchanging knowledge assets between workflow management systems, and tools for collaborative workflow development.

∇-Prox: Differentiable Proximal Algorithm Modeling for Large-Scale Optimization

Tasks across diverse application domains can be posed as large-scale optimization problems, these include graphics, vision, machine learning, imaging, health, scheduling, planning, and energy system forecasting. Independently of the application domain, proximal algorithms have emerged as a formal optimization method that successfully solves a wide array of existing problems, often exploiting problem-specific structures in the optimization. Although model-based formal optimization provides a principled approach to problem modeling with convergence guarantees, at first glance, this seems to be at odds with black-box deep learning methods. A recent line of work shows that, when combined with learning-based ingredients, model-based optimization methods are effective, interpretable, and allow for generalization to a wide spectrum of applications with little or no extra training data. However, experimenting with such hybrid approaches for different tasks by hand requires domain expertise in both proximal optimization and deep learning, which is often error-prone and time-consuming. Moreover, naively unrolling these iterative methods produces lengthy compute graphs, which when differentiated via autograd techniques results in exploding memory consumption, making batch-based training challenging. In this work, we introduce ∇-Prox, a domain-specific modeling language and compiler for large-scale optimization problems using differentiable proximal algorithms. ∇-Prox allows users to specify optimization objective functions of unknowns concisely at a high level, and intelligently compiles the problem into compute and memory-efficient differentiable solvers. One of the core features of ∇-Prox is its full differentiability, which supports hybrid model- and learning-based solvers integrating proximal optimization with neural network pipelines. Example applications of this methodology include learning-based priors and/or sample-dependent inner-loop optimization schedulers, learned with deep equilibrium learning or deep reinforcement learning. With a few lines of code, we show ∇-Prox can generate performant solvers for a range of image optimization problems, including end-to-end computational optics, image deraining, and compressive magnetic resonance imaging. We also demonstrate ∇-Prox can be used in a completely orthogonal application domain of energy system planning, an essential task in the energy crisis and the clean energy transition, where it outperforms state-of-the-art CVXPY and commercial Gurobi solvers.

Applying a model-driven approach to the development of power plant SCADA/EMS software

In this paper, we propose and apply a model-driven approach to develop applications that compose a suite known as Energy Management System (EMS). EMS is an essential tool for the operation of electrical systems. Although models of cyber–physical phenomena are extensively used in planning, designing, and operating power systems, the development of EMS applications often uses such models only as support and documentation, while the main artifact throughout the software development process is source code. In other words, the models are used merely to document and communicate some aspects of a system. The evolution of such a system is expensive and error-prone, requiring specialized personnel, skilled in both software engineering and power systems. Some of the problems arising from this manual process include: (i) substantial implementation effort in interpreting models and manually translating them into programs; (ii) greater risk of introducing software defects; (iii) risk of leaving documentation outdated, or implementation inconsistent with the intended design. Model-Driven Engineering (MDE) is an approach to constructing computing systems in which models are used as primary engineering artifacts, while programs (and other products) are automatically generated from the models, thus addressing the above limitations. MDE approaches have been adopted in several industrial areas, including smart grid automation, in which specialized modeling languages and tools facilitate the production of executable software from high-level models. For SCADA/EMS software development, however, a dedicated approach is still absent. We address these issues by proposing D-SPADES: a Model-Driven Engineering approach tailored to the EMS domain. D-SPADES defines a domain-specific modeling language based on block diagram notation, called EMSML, specifically targeted to describe the behavior of EMS applications. It also defines the associated mapping strategies and harnesses a set of tools for automatically transforming models into source code of applications that can be integrated into existing SCADA platforms. We have applied D-SPADES to model and automatically generate source code for two different applications from the Itaipu power plant: a voltage and reactive power controller and a component for a system-wide special protection scheme. Both applications were validated using a power system simulator and performed satisfactorily. The special protection scheme was deployed to the production system. Hence we demonstrate that D-SPADES is a viable approach to developing mission-critical applications.