Stack Overflow Research Articles

The use of generative AI for coding in academia

Abstract In a recent Perspective Piece, Cooper et al. (2024) highlight the benefits that large language models (LLM) bring to the coding process and offer some guidance around the use of LLM‐generated code in academic publications. Whilst we broadly support the stance taken by the authors, we offer a slightly different perspective from our experience of using both artificial intelligence (AI) generated code and more traditional resources such as Stack Overflow. We highlight how the integration of generative AI tools such as ChatGPT holds immense promise for advancing academic research, revolutionising coding education and practices, promoting inclusivity, and facilitating knowledge dissemination. However, we also acknowledge that researchers must navigate the ethical and methodological challenges inherent in harnessing AI technologies, prioritising transparency, validation, and critical inquiry to ensure the integrity and credibility of research initiatives.

Semantic Web Approaches in Stack Overflow

StackOverflow (SO), a prominent question-answering site for programming, has amassed a vast repository of user-generated content since its inception in 2008. This paper conducts a thorough analysis of research trends on SO, examining 170 publications from 2008 to 2019. Utilizing qualitative and quantitative methods, the study categorizes papers using literature review and Latent Dirichlet Allocation (LDA), identifying 62 topics grouped into 8 main categories. Additionally, it highlights tools developed by researchers using SO data sets, showcasing their practical applications. The analysis also identifies research gaps and proposes future directions for each research area. This study serves as a valuable resource for practitioners and researchers interested in utilizing community data sets, offering insights into existing work, essential tools and techniques, and potential avenues for future research.

"Math is a pain!": Understanding challenges and needs of the Machine Learning community on Stack Overflow

Stack Overflow (SO) is a widely recognized online question-and-answer platform for programming, which has also fostered a substantial community dedicated to machine learning (ML), providing a space for both novices and experts to exchange ideas and find solutions to ML-related problems. However, as a relative minority of this online programming platform, research has demonstrated lower engagement in the ML community, but it remains largely unexplored to understand what hinders the engagement and contribution from ML users' perspectives. This paper presents an empirical study based on 22 hours of semi-structured interviews and 131 survey responses with users on SO and reveals the key factors that may lead to the lower response rate and extended waiting time for ML questions on SO, which includes the unique quality requirement for posting ML questions, the discrepancy between time invested and benefits gained, the dispersed nature of the ML community across various platforms and the desired improvement for SO. Moreover, the qualitative study reveals a declining friendliness in SO's culture over time; the subsequent quantitative study corroborates that newcomers frequently encounter stress when posting and answering ML questions, even though this stress diminishes with increased experience. Additionally, we also explored the potential influence of generative AI tools (e.g., ChatGPT) on online question-and-answer platforms, specifically focusing on ML Q&A. We hope the results of this study can pave the way for enhancing the experience of ML users on online platforms, ultimately facilitating improved knowledge exchange and collaboration within the ML domain.

API comparison based on the non-functional information mined from Stack Overflow

When comparing similar APIs, developers tend to distinguish them from the aspects of functional details. At the same time, some important non-functional factors (such as performance, usability, and security) may be ignored or noticed after using the API in the project. This may result in unnecessary errors or extra costs. API-related questions are common on Stack Overflow, and they can give a well-rounded picture of the APIs. This provides us with a rich resource for API comparison. However, although many methods are offered for mining Questions and Answers (Q&As) automatically, they often suffer from two main problems: 1) they only focus on the functional information of APIs; 2) they analyze each text in isolation but ignore the correlations among them. In this paper, we propose an approach based on the pre-training model BERT to mine the non-functional information of APIs from Stack Overflow: we first tease out the correlations among questions, answers as well as corresponding reviews, so that one Q&A can be analyzed as a whole; then, an information extraction model is constructed by fine-tuning BERT with three subtasks—entity identification, aspect classification, and sentiment analysis separately, and we use it to mine the texts in Q&As step by step; finally, we summarize and visualize the results in a user-friendly way, so that developers can understand the information intuitively at the beginning of API selection. We evaluate our approach on 4,456 Q&As collected from Stack Overflow. The results show our approach can identify the correlations among reviews with 90.1% precision, and such information can improve the performance of the data mining process. In addition, the survey on maturers and novices indicates the understandability and helpfulness of our method. Moreover, compared with language models, our method can provide more intuitive and brief information for API comparison in non-functional aspects.

ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using LLMs

Technical Q&A sites are valuable for software developers seeking knowledge, but the code snippets they provide are often uncompilable and incomplete due to unresolved types and missing libraries. This poses a challenge for users who wish to reuse or analyze these snippets. Existing methods either do not focus on creating compilable code or have low success rates. To address this, we propose ZS4C, a lightweight approach for zero-shot synthesis of compilable code from incomplete snippets using Large Language Models (LLMs). ZS4C operates in two stages: first, it uses an LLM, like GPT-3.5, to identify missing import statements in a snippet; second, it collaborates with a validator (e.g., compiler) to fix compilation errors caused by incorrect imports and syntax issues. We evaluated ZS4C on the StatType-SO benchmark and a new dataset, Python-SO, which includes 539 Python snippets from Stack Overflow across the 20 most popular Python libraries. ZS4C significantly outperforms existing methods, improving the compilation rate from 63% to 95.1% compared to the state-of-the-art SnR, marking a 50.1% improvement. On average, ZS4C can infer more accurate import statements (with an F1 score of 0.98) than SnR, with an improvement of 8.5% in the F1.

A framework for generating recommendations based on trust in an informal e-learning environment

Rapid advancement in information technology promotes the growth of new online learning communities in an e-learning environment that overloads information and data sharing. When a new learner asks a question, how a system recommends the answer is the problem of the learner’s cold start. In this article, our contributions are: (i) We proposed a Trust-aware Deep Neural Recommendation (TDNR) framework that addresses learner cold-start issues in informal e-learning by modeling complex nonlinear relationships. (ii) We utilized latent Dirichlet allocation for tag modeling, assigning tag categories to newly posted questions and ranking experts related to specific tags for active questioners based on hub and authority scores. (iii) We enhanced recommendation accuracy in the TDNR model by introducing a degree of trust between questioners and responders. (iv) We incorporated the questioner-responder relational graph, derived from structural preference information, into our proposed model. We evaluated the proposed model on the Stack Overflow dataset using mean absolute precision (MAP), root mean squared error (RMSE), and F-measure metrics. Our significant findings are that TDNR is a hybrid approach that provides more accurate recommendations compared to rating-based and social-trust-based approaches, the proposed model can facilitate the formation of informal e-learning communities, and experiments show that TDNR outperforms the competing methods by an improved margin. The model’s robustness, demonstrated by superior MAE, RMSE, and F-measure metrics, makes it a reliable solution for addressing information overload and user sparsity in Stack Overflow. By accurately modeling complex relationships and incorporating trust degrees, TDNR provides more relevant and personalized recommendations, even in cold-start scenarios. This enhances user experience by facilitating the formation of supportive learning communities and ensuring new learners receive accurate recommendations.

An empirical study of developers’ challenges in implementing Workflows as Code: A case study on Apache Airflow

The Workflows as Code paradigm is becoming increasingly essential to streamline the design and management of complex processes within data-intensive software systems. These systems require robust capabilities to process, analyze, and extract insights from large datasets. Workflow orchestration platforms such as Apache Airflow are pivotal in meeting these needs, as they effectively support the implementation of the Workflows as Code paradigm. Nevertheless, despite its considerable advantages, developers still face challenges due to the specialized demands of workflow orchestration and the complexities of distributed execution environments. In this paper, we manually study 1000 sampled Stack Overflow posts derived from 9591 Airflow-related questions to understand developers’ challenges and root causes while implementing Workflows as Code. Our analysis results in a hierarchical taxonomy of Airflow-related challenges that contains 7 high-level categories and 14 sub-categories. We find that the most significant obstacles for developers arise when defining and executing their workflow. Our in-depth analysis identifies 10 root causes behind the challenges, including incorrect workflow configuration, complex environmental setup, and a lack of basic knowledge about Airflow and the external systems that it interacts with. Additionally, our analysis of references shared within the collected posts reveals that beyond the frequently cited Airflow documentation, documentation from external systems and third-party providers is also commonly referenced to address Airflow-related challenges.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Analysis of Software Developers' Programming Language Preferences and Community Behavior From Big5 Personality Traits

ABSTRACTMany programming languages and technologies have appeared for the purpose of software development. When choosing a programming language, the developers' cognitive attributes, such as the Big5 personality traits (BPT), may play a role. The developers' personality traits can be reflected in their social media content (e.g., tweets, statuses, Q&A, reputation). In this article, we predict the developers' programming language preferences (i.e., the pattern of picking up a language) from their BPT derived from their content produced on social media. We randomly collected data from a total of 820 Twitter (currently X) and Stack Overflow (SO) users. Then, we collected user features (i.e., BPT, word embedding of tweets) from Twitter and programming preferences (i.e., programming tags, reputation, question, answer) from SO. We applied various machine learning (ML) and deep learning (DL) techniques to predict their programming language preferences from their BPT. We also investigated other interesting insights, such as how reputation and question‐asking/replying are associated with the users' BPT. The findings suggest that developers with high openness, conscientiousness, and extraversion are inclined to mobile applications, object‐oriented programming, and web programming, respectively. Furthermore, developers with high openness and conscientiousness traits have a high reputation in the SO community. Our ML and DL techniques classify the developers' programming language preferences using their BPT with an average accuracy of 78%.

Validating pretrained language models for content quality classification with semantic-preserving metamorphic relations

Context:Utilizing pretrained language models (PLMs) has become common practice in maintaining the content quality of question-answering (Q&A) websites. However, evaluating the effectiveness of PLMs poses a challenge as they tend to provide local optima rather than global optima. Objective:In this study, we propose using semantic-preserving Metamorphic Relations (MRs) derived from Metamorphic Testing (MT) to address this challenge and validate PLMs. Methods:To validate four selected PLMs, we conducted an empirical experiment using a publicly available dataset comprising 60000 data points. We defined three groups of Metamorphic Relations (MRGs), consisting of thirteen semantic-preserving MRs, which were then employed to generate “Follow-up” testing datasets based on the original “Source” testing datasets. The PLMs were trained using a separate training dataset. A comparison was made between the predictions of the four trained PLMs for “Source” and “Follow-up” testing datasets in order to identify instances of violations, which corresponded to inconsistent predictions between the two datasets. If no violation was found, it indicated that the PLM was insensitive to the associate MR; thereby, the MR can be used for validation. In cases where no violation occurred across the entire MRG, non-violation regions were identified and supported simulation metamorphic testing. Results:The results of this study demonstrated that the proposed MRs could effectively serve as a validation tool for content quality classification on Stack Overflow Q&A using PLMs. One PLM did not violate the “Uppercase conversion” MRG and the “Duplication” MRG. Furthermore, the absence of violations in the MRGs allowed for the identification of non-violation regions, confirming the ability of the proposed MRs to support simulation metamorphic testing. Conclusion:The experimental findings indicate that the proposed MRs can validate PLMs effectively and support simulation metamorphic testing for PLMs. However, further investigations are required to enhance the semantic comprehension and common sense knowledge of PLMs and explore highly informative statistical patterns of PLMs, in order to improve their overall performance.

Communication Quality and the Cost of Language: Evidence from Stack Overflow

The transmission of information is crucial for productivity and growth. However, language differences may limit its effectiveness. This is particularly relevant for knowledge platforms that aim to be global, given the cultural diversity of the pool of users. In this paper, I empirically investigate how the exogenous cost of language affects communication quality and the trade-offs faced by knowledge platforms in implementing their website in multiple languages. I exploit the staggered introduction of websites for languages other than English on a question-and-answer platform to demonstrate two main facts. First, nonnative English speakers who contribute in English increase their answers’ quality once able to use their native language, and their answers are more likely to solve the questioner’s problem. The native-language answers drive the effect, which is larger when the question quality is higher and the incentives are stronger. Second, users who participate in their native language but not in English contribute lower-quality answers compared with those who contribute in English. This suggests that lower language barriers allow users with less expertise in the subject to participate. These results show that the platform should adopt multiple languages to maximise the quality of the information collected, although doing so may also result in an inflow of low-quality content from new users. This paper was accepted by Joshua Gans, business strategy. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.01333 .

Comparing emotions in ChatGPT answers and human answers to the coding questions on Stack Overflow.

Recent advances in generative Artificial Intelligence (AI) and Natural Language Processing (NLP) have led to the development of Large Language Models (LLMs) and AI-powered chatbots like ChatGPT, which have numerous practical applications. Notably, these models assist programmers with coding queries, debugging, solution suggestions, and providing guidance on software development tasks. Despite known issues with the accuracy of ChatGPT's responses, its comprehensive and articulate language continues to attract frequent use. This indicates potential for ChatGPT to support educators and serve as a virtual tutor for students. To explore this potential, we conducted a comprehensive analysis comparing the emotional content in responses from ChatGPT and human answers to 2000 questions sourced from Stack Overflow (SO). The emotional aspects of the answers were examined to understand how the emotional tone of AI responses compares to that of human responses. Our analysis revealed that ChatGPT's answers are generally more positive compared to human responses. In contrast, human answers often exhibit emotions such as anger and disgust. Significant differences were observed in emotional expressions between ChatGPT and human responses, particularly in the emotions of anger, disgust, and joy. Human responses displayed a broader emotional spectrum compared to ChatGPT, suggesting greater emotional variability among humans. The findings highlight a distinct emotional divergence between ChatGPT and human responses, with ChatGPT exhibiting a more uniformly positive tone and humans displaying a wider range of emotions. This variance underscores the need for further research into the role of emotional content in AI and human interactions, particularly in educational contexts where emotional nuances can impact learning and communication.

Exploring the problems, their causes and solutions of AI pair programming: A study on GitHub and Stack Overflow

With the recent advancement of Artificial Intelligence (AI) and Large Language Models (LLMs), AI-based code generation tools become a practical solution for software development. GitHub Copilot, the AI pair programmer, utilizes machine learning models trained on a large corpus of code snippets to generate code suggestions using natural language processing. Despite its popularity in software development, there is limited empirical evidence on the actual experiences of practitioners who work with Copilot. To this end, we conducted an empirical study to understand the problems that practitioners face when using Copilot, as well as their underlying causes and potential solutions. We collected data from 473 GitHub issues, 706 GitHub discussions, and 142 Stack Overflow posts. Our results reveal that (1) Operation Issue and Compatibility Issue are the most common problems faced by Copilot users, (2) Copilot Internal Error, Network Connection Error, and Editor/IDE Compatibility Issue are identified as the most frequent causes, and (3) Bug Fixed by Copilot, Modify Configuration/Setting, and Use Suitable Version are the predominant solutions. Based on the results, we discuss the potential areas of Copilot for enhancement, and provide the implications for the Copilot users, the Copilot team, and researchers.

Learning beyond books: A hybrid model to learn real‐world problems

AbstractThere are several initiatives underway to improve the learning of software developers. These attempts include the integration of GitHub into software engineering classes, the creation of learning management systems, gamification approaches, and collaborative learning platforms. These initiatives have demonstrated promise in boosting students' collaborative growth and cooperation abilities, emphasizing their potential influence on improving learning experiences in practical areas. Books, on the other hand, remain basic in education, but their physical size limits their ability to explore all practical elements of a topic in depth. This limitation requires more research and application of theoretical information in real‐world circumstances. In this work, we address the issue of limited space in traditional books that frequently prevents complete presentation of practical elements of a topic. To address this issue, we propose an application that improves the reading experience and accelerates the learning process. To anticipate themes, we use a combination of latent Dirichlet allocation (LDA) algorithms and a generative pre‐trained transformer. First, utilizing LDA to find potential topic keywords inside the text and then leveraging generative pretrained transformer to predict topic names based on the LDA produced keywords. In addition, a query builder module produces and executes queries depending on the current page's topic, obtaining real‐world issues from Stack Overflow. The system classifies results by query‐title similarity, question‐answer ranking, and content quality before displaying them to users. This bridges the gap between theoretical knowledge and practical application. We illustrate the usefulness of suggested tool using simulations, comparison with existing tools and user studies. The majority of users provide favorable comments and find it interesting and helpful for improving the learning process.

Automatic title completion for Stack Overflow posts and GitHub issues

Automatic title completion for Stack Overflow posts and GitHub issues

A Disruptive Research Playbook for Studying Disruptive Innovations

As researchers today, we are witnessing a fundamental change in our technologically-enabled world due to the advent and diffusion of highly disruptive technologies such as generative Artificial Intelligence (AI), Augmented Reality (AR) and Virtual Reality (VR). In particular, software engineering has been profoundly affected by the transformative power of disruptive innovations for decades, with a significant impact of technical advancements on social dynamics due to its socio-technical nature. In this paper, we reflect on the importance of formulating and addressing research problems in software engineering through a socio-technical lens, thus ensuring a holistic understanding of the complex phenomena in this field. We propose a research playbook with the aim of providing a guide to formulate compelling and socially relevant research questions and to identify the appropriate research strategies for empirical investigations, with an eye on the long-term implications of technologies or their use. We showcase how to apply the research playbook. Firstly, we show how it can be used retrospectively to reflect on a prior disruptive technology, Stack Overflow, and its impact on software development. Secondly, we show how it can be used to question the impact of two current disruptive technologies: AI and AR/VR. Finally, we introduce a specialized GPT model to support the researcher in framing future investigations. We conclude by discussing the broader implications of adopting the playbook for both researchers and practitioners in software engineering and beyond.

SCC-GPT: Source Code Classification Based on Generative Pre-Trained Transformers

Developers often rely on online resources, such as Stack Overflow (SO), to seek assistance for programming tasks. To facilitate effective search and resource discovery, manual tagging of questions and posts with the appropriate programming language is essential. However, accurate tagging is not consistently achieved, leading to the need for the automated classification of code snippets into the correct programming language as a tag. In this study, we introduce a novel approach to automated classification of code snippets from Stack Overflow (SO) posts into programming languages using generative pre-trained transformers (GPT). Our method, which does not require additional training on labeled data or dependency on pre-existing labels, classifies 224,107 code snippets into 19 programming languages. We employ the text-davinci-003 model of ChatGPT-3.5 and postprocess its responses to accurately identify the programming language. Our empirical evaluation demonstrates that our GPT-based model (SCC-GPT) significantly outperforms existing methods, achieving a median F1-score improvement that ranges from +6% to +31%. These findings underscore the effectiveness of SCC-GPT in enhancing code snippet classification, offering a cost-effective and efficient solution for developers who rely on SO for programming assistance.

What causes exceptions in machine learning applications? Mining machine learning-related stack traces on Stack Overflow

What causes exceptions in machine learning applications? Mining machine learning-related stack traces on Stack Overflow

Harmonising Contributions: Exploring Diversity in Software Engineering through CQA Mining on Stack Overflow

The need for collective intelligence in technology means that online Q&A platforms, such as Stack Overflow and Reddit, have become invaluable in building the global knowledge ecosystem. Despite literature demonstrating a prevalence of inclusion and contribution disparities in online communities, studies investigating the underlying reasons behind such fluctuations remain scarce. The current study examines Stack Overflow users’ contribution profiles, both in isolation and relative to various diversity metrics, including GDP and access to electricity. This study also examines whether such profiles propagate to the city and state levels, supplemented by granular data such as per capita income and education, before validating quantitative findings using content analysis. We selected 143 countries and compared the profiles of their respective users to assess implicit diversity-related complications that impact how users contribute. Results show that countries with high GDP, prominent R&D presence, less wealth inequality, and sufficient access to infrastructure tend to have more users, regardless of their development status. Similarly, cities and states where technology is more prevalent (e.g., San Francisco and New York) have more users who tend to contribute more often. Qualitative analysis reveals distinct communication styles based on users’ locations. Urban users exhibited assertive, solution-oriented behaviour, actively sharing information. Conversely, rural users engaged through inquiries and discussions, incorporating personal anecdotes, gratitude, and conciliatory language. Findings from this study may benefit scholars and practitioners, allowing them to develop sustainable mechanisms to bridge the inclusion and diversity gaps.

What Do Flutter Developers Ask About? An Empirical Study on Stack Overflow Posts

Since Google launched Flutter, an open-source framework, in 2017, many companies and software developers have turned to its use owing to its cross-platform feature. Other attractive features include hot reloading, a rich widget library, and improved performance compared to other cross-platform frameworks. Despite the rise in the use and adoption of the framework, little has been done to understand developers’ challenges. This study aims to understand what Flutter developers post on Stack Overflow – a popular Q&A website for developers. Analyzing such posts would help us understand the challenges faced by Flutter developers. To meet this goal, the study used a topic modeling approach to analyze all “flutter” tagged posts between 2019 and 2023. This study revealed state management, widgets, navigation, packages, and persistence as some areas developers face challenges. Further, the study established that there is a growth in the number of Flutter-related posts and developers. While the Flutter framework is promising for companies and software developers, this study points out areas where Flutter trainers and developers should emphasize. Flutter Framework developers should provide more documentation and support as the language matures.

An API Recommendation Method for Querying Mobile Computing Problems

Nowadays mobile computing has penetrated into our lives and work. Major companies have invested a lot of material and financial resources in the development of mobile terminals. For developers, querying the right API is extremely important. However, finding the right APIs can be time-consuming and laborious. In this paper, to solve the problems developers may face in the actual development process and improve the development efficiency, we propose RAMC (Recommendation APIs for Mobile Computing), an API recommendation approach leveraging word embedding technique and the information crawling from Stack Overflow posts and Java core packages, to recommend appropriate APIs for developers. Furthermore, RAMC also provides developers with label words, similar questions and relevant code. To evaluate the effectiveness of RAMC, we decided to analyze our system by simulating an instance. By testing a problem encountered during mobile computing development, the RAMC can effectively output the much-related API and tags for developers.