Effectiveness and efficiency of a domain-specific language for high-performance marine ecosystem simulation: a controlled experiment

Abstract

It is a long-standing hypothesis that the concise and customized notation of a DSL improves the performance of developers when compared with a GPL. For non-technical domains—e.g., science—, this hypothesis lacks empirical evidence. Given this lack of empirical evidence, we evaluate a DSL for ecological modeling designed and implemented by us with regard to performance improvements of developers as compared to a GPL. We conduct an online survey with embedded controlled experiments among ecologists to assess the correctness and time spent of the participants when using a DSL for ecosystem simulation specifications compared with a GPL-based solution. We observe that (1) solving tasks with the DSL, the participants’ correctness point score was —depending on the task— on average 61 % up to 63 % higher than with the GPL-based solution and their average time spent per task was reduced by 31 % up to 56 %; (2) the participants subjectively find it easier to work with the DSL, and (3) more than 90 % of the subjects are able to carry out basic maintenance tasks concerning the infrastructure of the DSL used in our treatment, which is based on another internal DSL embedded into Java. The tasks of our experiments are simplified and our web-based editor components do not offer full IDE-support. Our findings indicate that the development of further DSL for the specific needs of the ecological modeling community should be a worthwhile investment to increase its members’ productivity and to enhance the reliability of their scientific results.