Literate Programming Research Articles

Extending Jupyter with Multi-Paradigm Editors

Computational notebooks like the Jupyter programming environment have been popular, particularly for developing data-driven applications. One of its main benefits is that it easily supports different programming languages with exchangeable kernels. Thus, it makes the user interface of computational notebooks broadly accessible. While their literate programming paradigm has advantages, we can use this infrastructure to make other paradigms similarly easily and broadly accessible to developers. In our work, we demonstrate how the Jupyter infrastructure can be utilized with different interfaces for different programming paradigms, enabling even greater flexibility for programmers and making it easier for them to adopt different paradigms when they are most suitable. We present a prototype that adds graphical programming and a multi-paradigm editor on top of the Jupyter system. The multi-paradigm editor seamlessly combines the added graphical programming with the familiar notebook interface side-by-side, which can further help developers switch between programming paradigms when desired. A subsequent user evaluation demonstrates the benefits not only of alternate interfaces and paradigms but also of the flexibility of seamlessly switching between them. Finally, we discuss some of the challenges in implementing these systems and how these can enhance the software development process in the future.

Exploratory data science on supercomputers for quantum mechanical calculations

Literate programming—the bringing together of program code and natural language narratives—has become a ubiquitous approach in the realm of data science. This methodology is appealing as well for the domain of Density Functional Theory (DFT) calculations, particularly for interactively developing new methodologies and workflows. However, effective use of literate programming is hampered by old programming paradigms and the difficulties associated with using high performance computing (HPC) resources. Here we present two Python libraries that aim to remove these hurdles. First, we describe the PyBigDFT library, which can be used to setup materials or molecular systems and provides high-level access to the wavelet based BigDFT code. We then present the related remotemanager library, which is able to serialize and execute arbitrary Python functions on remote supercomputers. We show how together these libraries enable transparent access to HPC based DFT calculations and can serve as building blocks for rapid prototyping and data exploration.

AlphaPept: a modern and open framework for MS-based proteomics

In common with other omics technologies, mass spectrometry (MS)-based proteomics produces ever-increasing amounts of raw data, making efficient analysis a principal challenge. A plethora of different computational tools can process the MS data to derive peptide and protein identification and quantification. However, during the last years there has been dramatic progress in computer science, including collaboration tools that have transformed research and industry. To leverage these advances, we develop AlphaPept, a Python-based open-source framework for efficient processing of large high-resolution MS data sets. Numba for just-in-time compilation on CPU and GPU achieves hundred-fold speed improvements. AlphaPept uses the Python scientific stack of highly optimized packages, reducing the code base to domain-specific tasks while accessing the latest advances. We provide an easy on-ramp for community contributions through the concept of literate programming, implemented in Jupyter Notebooks. Large datasets can rapidly be processed as shown by the analysis of hundreds of proteomes in minutes per file, many-fold faster than acquisition. AlphaPept can be used to build automated processing pipelines with web-serving functionality and compatibility with downstream analysis tools. It provides easy access via one-click installation, a modular Python library for advanced users, and via an open GitHub repository for developers.

Teemi: An open-source literate programming approach for iterative design-build-test-learn cycles in bioengineering.

Synthetic biology dictates the data-driven engineering of biocatalysis, cellular functions, and organism behavior. Integral to synthetic biology is the aspiration to efficiently find, access, interoperate, and reuse high-quality data on genotype-phenotype relationships of native and engineered biosystems under FAIR principles, and from this facilitate forward-engineering strategies. However, biology is complex at the regulatory level, and noisy at the operational level, thus necessitating systematic and diligent data handling at all levels of the design, build, and test phases in order to maximize learning in the iterative design-build-test-learn engineering cycle. To enable user-friendly simulation, organization, and guidance for the engineering of biosystems, we have developed an open-source python-based computer-aided design and analysis platform operating under a literate programming user-interface hosted on Github. The platform is called teemi and is fully compliant with FAIR principles. In this study we apply teemi for i) designing and simulating bioengineering, ii) integrating and analyzing multivariate datasets, and iii) machine-learning for predictive engineering of metabolic pathway designs for production of a key precursor to medicinal alkaloids in yeast. The teemi platform is publicly available at PyPi and GitHub.

ATOP

The TEI XSL Stylesheets codebase is a tool that tackles the transformation of TEI XML documents to and from various formats. This includes the crucial activity of generating schemas from TEI ODD, the TEI XML–conformant specification format that allows one to write a schema language (for example, TEI P5 itself is written in ODD) or to customize TEI P5 using a literate programming approach. Because of the difficulties of maintaining this current set of stylesheets, a task force was created with the mission of developing, from scratch, an ODD processor that reads in a series of one or more TEI ODD customization files, merges them with a TEI language (likely, but not necessarily, TEI P5 itself), and generates RELAX NG and Schematron schemas. This paper presents a rationale for this task and the initial steps of this work.

The five pillars of computational reproducibility: bioinformatics and beyond.

Computational reproducibility is a simple premise in theory, but is difficult to achieve in practice. Building upon past efforts and proposals to maximize reproducibility and rigor in bioinformatics, we present a framework called the five pillars of reproducible computational research. These include (1) literate programming, (2) code version control and sharing, (3) compute environment control, (4) persistent data sharing and (5) documentation. These practices will ensure that computational research work can be reproduced quickly and easily, long into the future. This guide is designed for bioinformatics data analysts and bioinformaticians in training, but should be relevant to other domains of study.

Literate programming for motivating and teaching neural network-based approaches to solve differential equations

In this paper, we introduce novel instructional approaches to engage students in using modelling with data to motivate and teach differential equations. Specifically, we introduce a pedagogical framework that will execute instructional modules to teach different solution techniques for differential equations through repositories and notebook environments during real-time instruction. Each of these teaching modules employs a literate programming approach that uses the notebook environment to explain the concepts in anatural language, such as English, interspersed with snippets of macros and traditional source code on a web browser. The pedagogical approach employed is reproducible and leads to openaccess material for students to motivate and teach differential equations efficiently. We will share examples of this framework applied to teaching advanced concepts such as machine learning and neural network approaches for solving ordinary and partial differential equations as well as estimating parameters in these equations for given datasets. More details of the work can be accessed from https://aoguedao.github.io/teaching-ml-diffeq.

Teaching Data Science with Literate Programming Tools

This paper presents a case study on using Emacs and Org-mode for literate programming in undergraduate computer and data science courses. Over three academic terms, the author mandated these tools across courses in R, Python, C++, SQL, and more. The onboarding relied on simplified Emacs tutorials and starter configurations. Students gained proficiency after undertaking initial practice. Live coding sessions demonstrated the flexible instruction enabled by literate notebooks. Assignments and projects required documentation alongside functional code. Student feedback showed enthusiasm for learning a versatile IDE, despite some frustration with the learning curve. Skilled students highlighted efficiency gains in a unified environment. However, the uneven adoption of documentation practices pointed to a need for better incorporation into grading. Additionally, some students found Emacs unintuitive, desiring more accessible options. This highlights a need to match tools to skill levels, potentially starting novices with graphical IDEs before introducing Emacs. The key takeaways are as follows: literate programming aids comprehension but requires rigorous onboarding and reinforcement, and Emacs excels for advanced workflows but has a steep initial curve. With proper support, these tools show promise for data science education.

A Review of Containerization for Interactive and Reproducible Analysis

In recent decades the analysis of data has become increasingly computational. Correspondingly, this has changed how scientific and statistical work is shared. For example, it is now commonplace for underlying analysis code and data to be proffered alongside journal publications and conference talks. Unfortunately, sharing code faces several challenges. First, it is often difficult to take code from one computer and run it on another. Code configuration, version, and dependency issues often make this challenging. Secondly, even if the code runs, it is often hard to understand or interact with the analysis. This makes it difficult to assess the code and its findings, for example, in a peer review process. In this review we describe the combination of two computing technologies that help make analyses shareable, interactive, and completely reproducible. These technologies are (1) analysis containerization, which leverages virtualization to fully encapsulate analysis, data, code and dependencies into an interactive and shareable format, and (2) code notebooks, a literate programming format for interacting with analyses. The fusion of these two technologies offers significant advantages over using either individually. This review surveys how the combination enhances the accessibility and reproducibility of code, analyses, and ideas.

Maplib: Interactive, Literal RDF Model Mapping for Industry

Knowledge graphs are important for industrial digitalization. Industrial knowledge graphs are often mapped from multiple existing large data sources, and creating a mapping requires the time of scarce subject matter experts (SME). Interactive, literal programming for large scale mapping would allow mapping engineers to make good use of SME time, and document their work. Currently, there are no open source tools supporting such a process. To solve this problem, we implement maplib, which leverages existing tooling from data science. In data science, there is widespread use of literate programming using frameworks such as Jupyter notebooks to interactively prepare data and create analyses using in-memory tables called DataFrames. Maplib is implemented in Rust using Polars DataFrames and has Python bindings, allowing us to leverage tooling used in data science. Maplib implements the OTTR mapping language, which is highly suited for industrial use cases. Maplib features a SPARQL engine defined directly on DataFrames, making querying possible immediately after mapping. We evaluate our approach by comparing mapping and querying performance with Morph-KGC and SPARQL Anything on the GTFS Madrid benchmark. Our approach materializes the graph and is ready to query 47x-182x faster, and scales to models that are over twice as large. Morph-KGC and SPARQL Anything perform better for most, but not all of the queries once the graph has been constructed. On the end-to-end task of mapping and querying however, which is very important for interactive mapping, maplib performs better for most queries.

Context-sensitive parsing for programming languages

Parsing programming languages using context-sensitive rather than context-free grammars is being considered here because a stronger formalism might be beneficial for dealing with increasingly complex programming languages and their syntax, or is more appropriate in some applications. A new deterministic non-backtracking algorithm for parsing deterministic context-sensitive languages is described. It is a significant improvement of the algorithm built into WEAVE and CWEAVE tools for literate programming and requires that the language is described by a context-sensitive reduction system, namely a deterministic formalism similar to a context-sensitive grammar but with strict rules about how reductions are to be applied. The new algorithm uses a reduction automaton for finding the position of the next reduction at each step during parsing rather than a hardcoded trie that is build into the original algorithm of WEAVE and CWEAVE. The new algorithm performs at least twice as few operations per input symbol as the original one. Furthermore, it is shown that parsing a language described by a context-sensitive reduction system need not be limited to typesetting purposes as in literate programming but can be used as a general parsing approach.

HESS Opinions: Participatory Digital eARth Twin Hydrology systems (DARTHs) for everyone – a blueprint for hydrologists

Abstract. The “Digital Earth” (DE) metaphor is very useful for both end users and hydrological modelers (i.e., the coders). In this opinion paper, we analyze different categories of models with the view of making them part of Digital eARth Twin Hydrology systems (DARTHs). We stress the idea that DARTHs are not models, rather they are an appropriate infrastructure that hosts (certain types of) models and provides some basic services for connecting to input data. We also argue that a modeling-by-component strategy is the right one for accomplishing the requirements of the DE. Five technological steps are envisioned to move from the current state of the art of modeling. In step 1, models are decomposed into interacting modules with, for instance, the agnostic parts dealing with inputs and outputs separated from the model-specific parts that contain the algorithms. In steps 2 to 4, the appropriate software layers are added to gain transparent model execution in the cloud, independently of the hardware and the operating system of computer, without human intervention. Finally, step 5 allows models to be selected as if they were interchangeable with others without giving deceptive answers. This step includes the use of hypothesis testing, the inclusion of error of estimates, the adoption of literate programming and guidelines to obtain informative clean code. The urgency for DARTHs to be open source is supported here in light of the open-science movement and its ideas. Therefore, it is argued that DARTHs must promote a new participatory way of performing hydrological science, in which researchers can contribute cooperatively to characterize and control model outcomes in various territories. Finally, three enabling technologies are also discussed in the context of DARTHs – Earth observations (EOs), high-performance computing (HPC) and machine learning (ML) – as well as how these technologies can be integrated in the overall system to both boost the research activity of scientists and generate knowledge.

CROWDSOURCED ACOUSTIC OPEN DATA ANALYSIS WITH FOSS4G TOOLS

Abstract. NoiseCapture is an Android application developed by the Gustave Eiffel University and the French National Centre for Scientific Research as central element of a participatory approach to environmental noise mapping. The application is open-source, and all its data are available freely. This study presents the results of the first exploratory analysis of 3 years of data collection through the lens of sound sources. This analysis is only based on the tags given by the users and not on the sound spectrum of the measurement, which will be studied at a later stage. The first results are encouraging, we were able to observe well known temporal sound source dynamics like road sounds temporal dynamic related to commuting or bird songs in the dataset. We also found correlations between wind and rain tags and their measurements by the the national meteorological service. The context of the study, the Free and Open Source Software tools and techniques used and literate programming benefits are presented.

Five Guiding Principles to Make Jupyter Notebooks Fit for Earth Observation Data Education

There is a growing demand to train Earth Observation (EO) data users in how to access and use existing and upcoming data. A promising tool for data-related training is computational notebooks, which are interactive web applications that combine text, code and computational output. Here, we present the Learning Tool for Python (LTPy), which is a training course (based on Jupyter notebooks) on atmospheric composition data. LTPy consists of more than 70 notebooks and has taught over 1000 EO data users so far, whose feedback is overall positive. We adapted five guiding principles from different fields (mainly scientific computing and Jupyter notebook research) to make the Jupyter notebooks more educational and reusable. The Jupyter notebooks developed (i) follow the literate programming paradigm by a text/code ratio of 3, (ii) use instructional design elements to improve navigation and user experience, (iii) modularize functions to follow best practices for scientific computing, (iv) leverage the wider Jupyter ecosystem to make content accessible and (v) aim for being reproducible. We see two areas for future developments: first, to collect feedback and evaluate whether the instructional design elements proposed meet their objective; and second, to develop tools that automatize the implementation of best practices.

Open Source Software for Spatial Data Science

Much progress has been made in the development of software tools for spatial analysis since the special issue of Geographical Analysis appeared in 2006, devoted to “Recent advances in software for spatial analysis in the social sciences” (Rey and Anselin 2006). The 15 some years since the publication of the issue have been marked by major changes in the spatial analytical software landscape. Arguably, three important and somewhat related phenomena can be distinguished that drove these changes: the embedding of spatial analysis into spatial data science; the growing recognition of open science/open source principles in empirical work; and the increasing adoption of a literate programming perspective.

Literate programming with CCTBX and PyMOL in Jupyter notebooks

Literate programming with <i>CCTBX</i> and <i>PyMOL</i> in <i>Jupyter</i> notebooks

Genome-wide binding analysis of 195 DNA binding proteins reveals "reservoir" promoters and human specific SVA-repeat family regulation.

A key aspect in defining cell state is the complex choreography of DNA binding events in a given cell type, which in turn establishes a cell-specific gene-expression program. Here we wanted to take a deep analysis of DNA binding events and transcriptional output of a single cell state (K562 cells). To this end we re-analyzed 195 DNA binding proteins contained in ENCODE data. We used standardized analysis pipelines, containerization, and literate programming with R Markdown for reproducibility and rigor. Our approach validated many findings from previous independent studies, underscoring the importance of ENCODE's goals in providing these reproducible data resources. We also had several new findings including: (i) 1,362 promoters, which we refer to as 'reservoirs,' that are defined by having up to 111 different DNA binding-proteins localized on one promoter, yet do not have any expression of steady-state RNA (ii) Reservoirs do not overlap super-enhancer annotations and distinct have distinct properties from super-enhancers. (iii) The human specific SVA repeat element may have been co-opted for enhancer regulation and is highly transcribed in PRO-seq and RNA-seq. Collectively, this study performed by the students of a CU Boulder computational biology class (BCHM 5631 -Spring 2020) demonstrates the value of reproducible findings and how resources like ENCODE that prioritize data standards can foster new findings with existing data in a didactic environment.

Understanding and improving the quality and reproducibility of Jupyter notebooks.

Jupyter Notebooks have been widely adopted by many different communities, both in science and industry. They support the creation of literate programming documents that combine code, text, and execution results with visualizations and other rich media. The self-documenting aspects and the ability to reproduce results have been touted as significant benefits of notebooks. At the same time, there has been growing criticism that the way in which notebooks are being used leads to unexpected behavior, encourages poor coding practices, and makes it hard to reproduce its results. To better understand good and bad practices used in the development of real notebooks, in prior work we studied 1.4 million notebooks from GitHub. We presented a detailed analysis of their characteristics that impact reproducibility, proposed best practices that can improve the reproducibility, and discussed open challenges that require further research and development. In this paper, we extended the analysis in four different ways to validate the hypothesis uncovered in our original study. First, we separated a group of popular notebooks to check whether notebooks that get more attention have more quality and reproducibility capabilities. Second, we sampled notebooks from the full dataset for an in-depth qualitative analysis of what constitutes the dataset and which features they have. Third, we conducted a more detailed analysis by isolating library dependencies and testing different execution orders. We report how these factors impact the reproducibility rates. Finally, we mined association rules from the notebooks. We discuss patterns we discovered, which provide additional insights into notebook reproducibility. Based on our findings and best practices we proposed, we designed Julynter, a Jupyter Lab extension that identifies potential issues in notebooks and suggests modifications that improve their reproducibility. We evaluate Julynter with a remote user experiment with the goal of assessing Julynter recommendations and usability.

Practical Reproducibility in Geography and Geosciences

Reproducible research is often perceived as a technological challenge, but it is rooted in the challenge to improve scholarly communication in an age of digitization. When computers become involved and researchers want to allow other scientists to inspect, understand, evaluate, and build on their work, they need to create a research compendium that includes the code, data, computing environment, and script-based workflows used. Here, we present the state of the art for approaches to reach this degree of computational reproducibility, addressing literate programming and containerization while paying attention to working with geospatial data (digital maps, geographic information systems). We argue that all researchers working with computers should understand these technologies to control their computing environment, and we present the benefits of reproducible workflows in practice. Example research compendia illustrate the presented concepts and are the basis for challenges specific to geography and geosciences. Based on existing surveys and best practices from different scientific domains, we conclude that researchers today can overcome many barriers and achieve a very high degree of reproducibility. If the geography and geosciences communities adopt reproducibility and the underlying technologies in practice and in policies, they can transform the way researchers conduct and communicate their work toward increased transparency, understandability, openness, trust, productivity, and innovation.

Software documentation with markdoc 5.0

markdoc is a general-purpose literate programming package for generating dynamic documents, dynamic presentation slides, Stata help files, and package vignettes in various formats. In this article, I introduce markdoc version 5.0, which performs independently of any third-party software, using the mini engine. The mini engine is a lightweight alternative to Pandoc (MacFarlane [2006, https://pandoc.org/ ]), completely written in Stata. I also propose a procedure for remodeling package documentation and data documentation in Stata and present a tutorial for generating help files, package vignettes, and GitHub Wiki documentation using markdoc.