Abstract
Jupyter notebooks has emerged as a standard tool for data science programming. Programs in Jupyter notebooks are different from typical programs as they are constructed by a collection of code snippets interleaved with text and visualisation. This allows interactive exploration and snippets may be executed in different order which may give rise to different results due to side-effects between snippets. Previous studies have shown the presence of considerable code duplication -- code clones -- in sources of traditional programs, in both so-called systems programming languages and so-called scripting languages. In this paper we present the first large-scale study of code cloning in Jupyter notebooks. We analyse a corpus of 2.7 million Jupyter notebooks hosted on GitHJub, representing 37 million individual snippets and 227 million lines of code. We study clones at the level of individual snippets, and study the extent to which snippets are recurring across multiple notebooks. We study both identical clones and approximate clones and conduct a small-scale ocular inspection of the most common clones. We find that code cloning is common in Jupyter notebooks -- more than 70% of all code snippets are exact copies of other snippets (with possible differences in white spaces), and around 50% of all notebooks do not have any unique snippet, but consists solely of snippets that are also found elsewhere. In notebooks written in Python, at least 80% of all snippets are approximate clones and the prevalence of code cloning is higher in Python than in other languages. We further find that clones between different repositories are far more common than clones within the same repository. However, the most common individual repository from which a Jupyter notebook contains clones is the repository in which itself resides.
Highlights
Data science, that is processing, analysing and extracting knowledge from large quantities of data, has emerged as a new inter-disciplinary field or new research paradigm, and an increasingly important component in industry, as many companies strive to be “data-driven”
In this paper we present the first large-scale study of code cloning in Jupyter notebooks
We find that code cloning is common in Jupyter notebooks – more than 70 % of all code snippets are exact copies of other snippets, and around 50 % of all notebooks do not have any unique snippet, but consists solely of snippets that are found elsewhere
Summary
That is processing, analysing and extracting knowledge from large quantities of data, has emerged as a new inter-disciplinary field or new research paradigm, and an increasingly important component in industry, as many companies strive to be “data-driven”. The emergence and rapid growth of this field is fuelled by the availability and easy access to vast quantities of data, the relative ease with which such data sets can be gathered with new technology, and the availability of easyto-use computational tools that hide most of the complicated data crunching and computation behind (relatively speaking) easy interfaces. This allows a new class of programmers — that would not traditionally view themselves as such — to explore data sets and use statistical methods for business decisions, in research, and in society. Jupyter Notebook files are stored on disk in JSON format
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
