GitHub Issues Research Articles

Exploring the problems, their causes and solutions of AI pair programming: A study on GitHub and Stack Overflow

With the recent advancement of Artificial Intelligence (AI) and Large Language Models (LLMs), AI-based code generation tools become a practical solution for software development. GitHub Copilot, the AI pair programmer, utilizes machine learning models trained on a large corpus of code snippets to generate code suggestions using natural language processing. Despite its popularity in software development, there is limited empirical evidence on the actual experiences of practitioners who work with Copilot. To this end, we conducted an empirical study to understand the problems that practitioners face when using Copilot, as well as their underlying causes and potential solutions. We collected data from 473 GitHub issues, 706 GitHub discussions, and 142 Stack Overflow posts. Our results reveal that (1) Operation Issue and Compatibility Issue are the most common problems faced by Copilot users, (2) Copilot Internal Error, Network Connection Error, and Editor/IDE Compatibility Issue are identified as the most frequent causes, and (3) Bug Fixed by Copilot, Modify Configuration/Setting, and Use Suitable Version are the predominant solutions. Based on the results, we discuss the potential areas of Copilot for enhancement, and provide the implications for the Copilot users, the Copilot team, and researchers.

Facilitating pathway and network based analysis of RNA-Seq data with pathlinkR.

R package pathlinkR is designed to aid transcriptomic analyses by streamlining and simplifying the process of analyzing and interpreting differentially expressed genes derived from human RNA-Seq data. It provides an integrated approach to performing pathway enrichment and network-based analyses, while also producing publication-quality figures to summarize these results, allowing users to more efficiently interpret their findings and extract biological meaning from large amounts of data. pathlinkR is available to install from the software repository Bioconductor at https://bioconductor.org/packages/pathlinkR/, with support available through the Bioconductor forums. The code, example, and supporting data is available on the GitHub repository at https://github.com/hancockinformatics/pathlinkR, under the GPL-3.0 license, where users may report problems or make suggestions using GitHub's issue system.

Automatic title completion for Stack Overflow posts and GitHub issues

Automatic title completion for Stack Overflow posts and GitHub issues

Can GitHub Issues Help in App Review Classifications?

Can GitHub Issues Help in App Review Classifications?

Forum Text Processing and Summarization

Frequently Asked Questions (FAQs) are extensively studied in general domains like the medical field, but such frameworks are lacking in domains such as software engineering and open-source communities. This research aims to bridge this gap by establishing the foundations of an automated FAQ Generation and Retrieval framework specifically tailored to the software engineering domain. The framework involves analyzing, ranking, performing sentiment analysis, and summarization techniques on open forums like StackOverflow and GitHub issues. A corpus of Stack Overflow post data is collected to evaluate the proposed framework and the selected models. Integrating state-of-the-art models of string-matching models, sentiment analysis models, summarization models, and the proprietary ranking formula proposed in this paper forms a robust Automatic FAQ Generation and Retrieval framework to facilitate developers' work. String matching, sentiment analysis, and summarization models are evaluated, and F1 scores of 71.31%, 74.90%, and 53.4% were achieved. Given the subjective nature of evaluations in this context, a human review is used to further validate the effectiveness of the overall framework, with assessments made on relevancy, preferred ranking, and preferred summarization. Future work includes improving summarization models by incorporating text classification and summarizing them individually (Kou et al, 2023), as well as proposing feedback loop systems based on human reinforcement learning. Furthermore, efforts will be made to optimize the framework by utilizing knowledge graphs for dimension reduction, enabling it to handle larger corpora effectively

The Human Side of Fuzzing: Challenges Faced by Developers during Fuzzing Activities

Fuzz testing, also known as fuzzing, is a software testing technique aimed at identifying software vulnerabilities. In recent decades, fuzzing has gained increasing popularity in the research community. However, existing studies led by fuzzing experts mainly focus on improving the coverage and performance of fuzzing techniques. That is, there is still a gap in empirical knowledge regarding fuzzing, especially about the challenges developers face when they adopt fuzzing. Understanding these challenges can provide valuable insights to both practitioners and researchers on how to further improve fuzzing processes and techniques. We conducted a study to understand the challenges encountered by developers during fuzzing. More specifically, we first manually analyzed 829 randomly sampled fuzzing-related GitHub issues and constructed a taxonomy consisting of 39 types of challenges (22 related to the fuzzing process itself, 17 related to using external fuzzing providers). We then surveyed 106 fuzzing practitioners to verify the validity of our taxonomy and collected feedback on how the fuzzing process can be improved. Our taxonomy, accompanied with representative examples and highlighted implications, can serve as a reference point on how to better adopt fuzzing techniques for practitioners, and indicates potential directions researchers can work on toward better fuzzing approaches and practices.

Detecting outdated code element references in software repository documentation

Outdated documentation is a pervasive problem in software development, preventing effective use of software, and misleading users and developers alike. We posit that one possible reason why documentation becomes out of sync so easily is that developers are unaware of when their source code modifications render the documentation obsolete. Ensuring that the documentation is always in sync with the source code takes considerable effort, especially for large codebases. To address this situation, we propose an approach that can automatically detect code element references that survive in the documentation after all source code instances have been deleted. In this work, we analysed over 3,000 GitHub projects and found that most projects contain at least one outdated code element reference at some point in their history. We submitted GitHub issues to real-world projects containing outdated references detected by our approach, some of which have already led to documentation fixes. As an initiative toward keeping documentation in software repositories up-to-date, we have made our implementation available for developers to scan their GitHub projects for outdated code element references.

Automatic Core-Developer Identification on GitHub: A Validation Study

Many open-source software projects are self-organized and do not maintain official lists with information on developer roles. So, knowing which developers take core and maintainer roles is, despite being relevant, often tacit knowledge. We propose a method to automatically identify core developers based on role permissions of privileged events triggered in GitHub issues and pull requests. In an empirical study on 25/GitHub projects, (1) we validate the set of automatically identified core developers with a sample of project-reported developer lists, and (2) we use our set of identified core developers to assess the accuracy of state-of-the-art unsupervised developer classification methods. Our results indicate that the set of core developers, which we extracted from privileged issue events, is sound and the accuracy of state-of-the-art unsupervised classification methods depends mainly on the data source (commit data versus issue data) rather than the network-construction method (directed versus undirected, etc.). In perspective, our results shall guide research and practice to choose appropriate unsupervised classification methods, and our method can help create reliable ground-truth data for training supervised classification methods.

Promoting High-Quality Data in OBIS: Insights from the OBIS Data Quality Assessment and Enhancement Project Team

The Ocean Biodiversity Information System (OBIS) (Klein et al. 2019) is a global database of marine biodiversity and associated environmental data, which provides critical information to researchers and policymakers worldwide. Ensuring the accuracy and consistency of the data in OBIS is essential for its usefulness and value, not only to the scientific community but also to the science-policy interface. The OBIS Data Quality Assessment and Enhancement Project Team (QCPT), formed in 2019 by the OBIS steering group, aims to assess and enhance data quality. It has been working on three categories of activities for this purpose: Data quality enhancement and management The OBIS QCPT organized data laundry events to identify and address data quality issues of published OBIS datasets. Furthermore, individual OBIS nodes were invited to give their data-processing presentations in the monthly meetings to foster knowledge sharing and collaborative problem-solving focused on data quality. Data quality issues and solutions highlighted in the presentations and data laundry events were documented in a dedicated GitHub repository as GitHub issues. The solutions for data quality issues and marine-specific pre-publication quality control tools, designed to identify the data quality issues, were provided as feedback to the OBIS Capacity Development Task Team. These inputs were used to create training resources (see OBIS manual, upcoming OBIS training course hosted on OceanTeacher Global Academy) aimed at preventing these issues. Standardization of OBIS data processing pipeline As OBIS uses the Darwin Core standard (Wieczorek et al. 2012), the use of standardized tests and assertions in the data processing pipeline is encouraged. To achieve this, the OBIS QCPT aligned OBIS quality checks with a subset of core tests and assertions (Chapman et al. 2020) developed by the Biodiversity Information Standards (TDWG) Biodiversity Data Quality (BDQ) Task Group 2 (TG2) (Chapman et al. 2020) as tracked in this GitHub issue. Not all default parameters of the core tests and assertions are optimal for marine biodiversity data. The OBIS QCPT met monthly to determine suitable parameters for customizing the tests. The pipeline produces a data quality report for each dataset with quality flags that indicate potential data quality issues, enabling node managers and data providers to review the flagged records. Community engagement The OBIS QCPT led a survey among data users to gather insights into OBIS data quality issues and bridge the gap between the current implementation and user expectations. The survey findings enabled OBIS to prioritize issues to be addressed, as summarized in Section 2.2.2 of the 11th OBIS Steering Group meeting report. In addition to engaging with data users, the OBIS QCPT also served as a platform to discuss questions related to the use of Darwin Core from the nodes and provided feedback for the term discussions. In summary, the OBIS QCPT improves marine species data reliability and usability through transparent and participatory approaches, fostering continuous improvement. Collaborative efforts, standardized procedures, and knowledge sharing advance OBIS' mission of providing high quality biodiversity data for research, conservation, and ocean management.

Automating discussion structure re-organization for GitHub issues

Automating discussion structure re-organization for GitHub issues

Adopting Two Supervisors for Efficient Use of Large-Scale Remote Deep Neural Networks

Recent decades have seen the rise of large-scale Deep Neural Networks (DNNs) to achieve human-competitive performance in a variety of artificial intelligence tasks. Often consisting of hundreds of millions, if not hundreds of billion parameters, these DNNs are too large to be deployed to, or efficiently run on resource-constrained devices such as mobile phones or IoT microcontrollers. Systems relying on large-scale DNNs thus have to call the corresponding model over the network, leading to substantial costs for hosting and running the large-scale remote model, costs which are often charged on a per-use basis. In this paper, we propose BiSupervised , a novel architecture, where, before relying on a large remote DNN, a system attempts to make a prediction on a small-scale local model. A DNN supervisor monitors said prediction process and identifies easy inputs for which the local prediction can be trusted. For these inputs, the remote model does not have to be invoked, thus saving costs, while only marginally impacting the overall system accuracy. Our architecture furthermore foresees a second supervisor to monitor the remote predictions and identify inputs for which not even these can be trusted, allowing to raise an exception or run a fallback strategy instead. We evaluate the cost savings, and the ability to detect incorrectly predicted inputs on four diverse case studies: IMDB movie review sentiment classification, Github issue triaging, Imagenet image classification, and SQuADv2 free-text question answering. In all four case studies, we find that BiSupervised allows to reduce cost by at least 30% while maintaining a similar system-level prediction performance. In two case studies (IMDB and SQuADv2), we find that BiSupervised even achieves a higher system-level accuracy, at reduced cost, compared to a remote-only model. Furthermore, measurements taken on our setup indicate a large potential of BiSupervised to reduce average prediction latency.

Improving Access to Collection Information in Asia and Africa through the Global Registry of Scientific Collections

Global Registry of Scientific Collections, or GRSciColl, is a comprehensive registry of natural history collections maintained by the Global Biodiversity Information Facility (GBIF), building upon previous efforts developed by the Consortium of the Barcode of Life (CBOL) and other organizations. Its primary purpose is to provide a centralized platform for accessing information on institutions, their collections and associated staff members. While both GBIF and GRSciColl aim to encompass data from all around the globe, Asian and African collections are often underrepresented. To address this disparity, initiatives like the Biodiversity Information for Development (BID) and the Biodiversity Information Fund for Asia (BIFA) have been established. These initiatives specifically focus on assisting institutions from underrepresented regions in sharing their data through GBIF. In addition to providing support in the framework of these initiatives, the GBIF Regional support teams are actively engaged in efforts to improve access to relevant data about Asian and African collections. This involves updating information on Asian and African institutions and their collections in GRSciColl. Furthermore, the teams assist towards publishing biodiversity data held by these institutions for open access. To support this objective, a Collection Mobilization repository has been established on GitHub, with more than 1,900 issues created at an institutional level and a systematic workflow is developed and followed. The overall aim of this workflow adheres to the goal of GRSciColl in providing an updated information on scientific collections, their host institutions, and relevant staff, also making the specimen data from those institutions open access when possible. Here are a few key points of this ongoing work: The process begins by compiling relevant information on institutions and their collections by browsing the institution’s web pages to gather accurate and up-to-date data. Based on the findings, necessary changes or additions to the existing data are made in the institution and collection page of the GRSciColl registry, and simultaneously noted in related GitHub issues in the collection mobilization repository, ensuring transparency and record keeping. When required, contact is established with the staff of the institution, to seek clarifications and ensure accuracy of the information in the registry. Appropriate GitHub labels are then assigned to the institutions based on their data mobilization or digitization status, to help prioritize the ongoing work. Institutions that are familiar with GBIF data infrastructure or those that have digitized records of the specimens are given priority for contact to gauge their interest in mobilizing their data for open access. Institutions interested in publishing their data are offered support in the process, and their datasets are added to the Data Mobilization Repository in GitHub as an issue, using the Suggest a Dataset tool on GBIF.org. Finally, progress on data preparation and then publication is tracked and relevant comments and labels are added in the GitHub issues. The process begins by compiling relevant information on institutions and their collections by browsing the institution’s web pages to gather accurate and up-to-date data. Based on the findings, necessary changes or additions to the existing data are made in the institution and collection page of the GRSciColl registry, and simultaneously noted in related GitHub issues in the collection mobilization repository, ensuring transparency and record keeping. When required, contact is established with the staff of the institution, to seek clarifications and ensure accuracy of the information in the registry. Appropriate GitHub labels are then assigned to the institutions based on their data mobilization or digitization status, to help prioritize the ongoing work. Institutions that are familiar with GBIF data infrastructure or those that have digitized records of the specimens are given priority for contact to gauge their interest in mobilizing their data for open access. Institutions interested in publishing their data are offered support in the process, and their datasets are added to the Data Mobilization Repository in GitHub as an issue, using the Suggest a Dataset tool on GBIF.org. Finally, progress on data preparation and then publication is tracked and relevant comments and labels are added in the GitHub issues. The ongoing work ensures continuous improvement and expansion of GRSciColl, while promoting open access to valuable scientific data. This presentation will highlight the work done so far, as well as the challenges and lessons learnt from this ongoing project.

Gender Diversity on GitHub Issue Tracking: What's the Difference?

This work analyzes female participation in communication on GitHub’s Issue Tracking, based on thematic relevance of posted comments according to the developer’s gender relative to other metrics, such as reputation, participation time on the platform, and number of reported issues. Data from 5 open source communities and 5 communities dedicated to women was analyzed. The results indicate that, on average, the relevance of comments made by women is similar to that of men. However, the study confirms other findings in literature that highlight low levels of female representativeness and participation in projects, with just 22% of comments posted by women and 16% of issues reported by them.

CrossFix: Resolution of GitHub issues via similar bugs recommendation

SummaryWith the increasing popularity of Open‐Source Software (OSS), the number of GitHub issues reported daily in these OSS projects has been growing rapidly. To resolve these issues, developers need to spend time and effort in debugging and fixing these issues. Meanwhile, a recent approach shows that similar bugs exist across different projects, and one could use the GitHub issues from a different project for finding new bugs for a related project. To locate similar bugs for our approach, we first conduct a study of similar bugs in GitHub. Our study redefines similar bugs as bugs that share the (1) same libraries, (2) same functionalities, (3) same reproduction steps, (4) same configurations, (5) same outcomes, or (6) same errors. Moreover, our study revealed the usefulness of similar bugs in helping developers to find more contexts about the bug and fixing. Based on our study, we design CrossFix, a tool that automatically suggests relevant GitHub issues based on an open GitHub issue. The suggested GitHub issues may contain solutions written in natural language or pull requests that help developers in resolving the given issue. Our evaluation on 249 open issues from Java and Android projects shows that CrossFix could suggest similar bugs to help developers in debugging and fixing.

CCPlotR: an R package for the visualization of cell-cell interactions.

We present CCPlotR-an R package that generates visualizations of cell-cell interactions. CCPlotR is designed to work with the output of tools that predict cell-cell interactions from single-cell gene expression data and requires only a table of predicted interactions as input. The package can generate a comprehensive set of publication-ready figures such as heatmaps, dotplots, circos plots and network diagrams, providing a useful resource for researchers working on cell-cell interactions. CCPlotR is available to download and install from GitHub (https://github.com/Sarah145/CCPlotR) and comes with a toy dataset to demonstrate the different functions. Support for users will be provided via the GitHub issues tracker (https://github.com/Sarah145/CCPlotR/issues).

An empirical study of security practices for microservices systems

Despite the numerous benefits of microservices systems, security has been a critical issue in such systems. Several factors explain this difficulty, including a knowledge gap among microservices practitioners on properly securing a microservices system. To (partially) bridge this gap, we conducted an empirical study. We first manually analyzed 861 microservices security points, including 567 issues, 9 documents, and 3 wiki pages from 10 GitHub open-source microservices systems and 306 Stack Overflow posts concerning security in microservices systems. In this study, a microservices security point is referred to as “a GitHub issue, a Stack Overflow post, a document, or a wiki page that entails 5 or more microservices security paragraphs”. Our analysis led to a catalog of 28 microservices security practices. We then ran a survey with 74 microservices practitioners to evaluate the usefulness of these 28 practices. Our findings demonstrate that the survey respondents affirmed the usefulness of the 28 practices. We believe that the catalog of microservices security practices can serve as a valuable resource for microservices practitioners to more effectively address security issues in microservices systems. It can also inform the research community of the required or less explored areas to develop microservices-specific security practices and tools.

Maintaining Values

Communication technologies have significant social impacts, and it is important to consider how designers' and developers' values shape their design. Increasingly, these technologies are released as continually evolving platforms and services, so their development involves ongoing discussions and debates about unforeseen problems and future directions. However, there is a gap in research about how designers, developers, and other stakeholders engage with values during later stages of development. We investigate discussions about values in the context of open source software development, focusing on projects related to the Decentralized Web. We conducted a large-scale analysis of GitHub issues among diverse yet ideologically-related projects. We show that the percentage of discussions about values increases later in development, and we identify features and outcomes of conflicts related to open source participants' values. Finally, we propose suggestions to improve upon existing discussion practices by supporting common ground among collaborators with diverse goals, perspectives, and experiences.

Detecting non-natural language artifacts for de-noising bug reports

Textual documents produced in the software engineering process are a popular target for natural language processing (NLP) and information retrieval (IR) approaches. However, issue tickets often contain artifacts such as code snippets, log outputs and stack traces. These artifacts not only inflate the issue ticket sizes, but also can this noise constitute a real problem for some NLP approaches, and therefore has to be removed in the pre-processing of some approaches. In this paper, we present a machine learning based approach to classify textual content into natural language and non-natural language artifacts at line level. We show how data from GitHub issue trackers can be used for automated training set generation, and present a custom preprocessing approach for the task of artifact removal. The training sets are automatically created from Markdown annotated issue tickets and project documentation files. We use these generated training sets to train a Markdown agnostic model that is able to classify un-annotated content. We evaluate our approach on issue tickets from projects written in C++, Java, JavaScript, PHP, and Python. Our approach achieves ROC-AUC scores between 0.92 and 0.96 for language-specific models. A multi-language model trained on the issue tickets of all languages achieves ROC-AUC scores between 0.92 and 0.95. The provided models are intended to be used as noise reduction pre-processing steps for NLP and IR approaches working on issue tickets.

The Benefits of an Open-Science Approach in Student Research Projects

Student projects can sometimes be overlooked when considering open science methods. This is perhaps because of the limited time available to complete the project, a lack of confidence in their open research skills by supervision staff or students, and lower expectations of publications resulting from student project work (particularly at undergraduate level). However, open science methods add value to student projects and provide benefits on several levels.
 This talk describes from both the student and supervisor perspectives how open science can benefit student projects - from methods to management - using recent undergraduate projects as an example. We give a brief description of the concept of open science as we understand it and discuss the general benefits of an open science approach to research.
 The Edward Wallace lab (School of Biological Sciences, University of Edinburgh) has run several student projects working with and extending the open source software tool 'riboviz' to process and analyse ribosome profiling sequencing datasets. This collaborative project worked with experts in the UK and USA and aimed to add new features and capabilities to riboviz, thoroughly documenting the software, and successfully using it to analyse a diverse range of datasets from organisms across the tree of life.
 Student projects recreated existing published results using riboviz, developed new visualisation scripts, and added new optional features to the codebase to address specific research questions.
 This talk discusses how both the student project and the riboviz project fit into open science research, covering how open science tools were used to plan, manage and carry out an open science project and the challenges these presented. For example, as this was a computer-based project, we discuss the tools and skills needed by the students to successfully complete the project, including programming skills (Unix command-line, R, Python) and version control (using Git and GitHub) and the open resources used to learn and develop these skills.
 Open software tools such as version control are also explained in the context of student supervision and project management (for example how to use GitHub issue tickets for identifying and quickly solving project problems remotely), and for writing collaborative manuscripts (such as the recently published riboviz 2 paper and open science article).
 Our experiences provide a strong case study for successful open science approaches in student projects and clear evidence of their long-term benefits to students, supervisors and wider lab groups. We highlight our "top takeaways" aimed at students, day-to-day supervisors, and PIs and demonstrate how others can take the same approach.

MULA: A Just-In-Time Multi-labeling System for Issue Reports

A very important function of an issue tracking system is to assign labels to issue reports, such as bug, feature, enhancement, etc., in order to categorize issues to facilitate various development activities. In practice, it is very common that an issue has multiple labels. However, current works are mainly based on single-label prediction, which are not suitable for just-in-time multi-labeling services, due to the low efficiency. Therefore, in this paper, we propose MULA, a just-in-time MUlti-LAbeling system, which learns and automatically assigns multiple labels to issue reports. We have built a dataset with 81,601 entries and 11 labels, as the first benchmark for this task, and implemented a GitHub app. To the best of our knowledge, this is the first work and tool for online multi-labeling GitHub issues based on their categories. We conduct a comprehensive empirical study, including comparisons with five commonly adopted labeling models that show the superiority of MULA, as well as an evaluation that shows high consistency between MULA’s suggestions and developers’ opinions.