Mining Software Repositories Research Articles

A survey on the use of topic models when mining software repositories

Researchers in software engineering have attempted to improve software development by mining and analyzing software repositories. Since the majority of the software engineering data is unstructured...

Lessons learned from applying social network analysis on an industrial Free/Libre/Open Source Software ecosystem

Many software projects are no longer done in-house by a single organization. Instead, we are in a new age where software is developed by a networked community of individuals and organizations, which base their relations to each other on mutual interest. Paradoxically, recent research suggests that software development can actually be jointly-developed by rival firms. For instance, it is known that the mobile-device makers Apple and Samsung kept collaborating in open source projects while running expensive patent wars in the court. Taking a case study approach, we explore how rival firms collaborate in the open source arena by employing a multi-method approach that combines qualitative analysis of archival data (QA) with mining software repositories (MSR) and Social Network Analysis (SNA). While exploring collaborative processes within the OpenStack ecosystem, our research contributes to Software Engineering research by exploring the role of groups, sub-communities and business models within a high-networked open source ecosystem. Surprising results point out that competition for the same revenue model (i.e., operating conflicting business models) does not necessary affect collaboration within the ecosystem. Moreover, while detecting the different sub-communities of the OpenStack community, we found out that the expected social tendency of developers to work with developers from same firm (i.e., homophily) did not hold within the OpenStack ecosystem. Furthermore, while addressing a novel, complex and unexplored open source case, this research also contributes to the management literature in coopetition strategy and high-tech entrepreneurship with a rich description on how heterogeneous actors within a high-networked ecosystem (involving individuals, startups, established firms and public organizations) joint-develop a complex infrastructure for big-data in the open source arena.

MSR4SM: Using topic models to effectively mining software repositories for software maintenance tasks

ContextMining software repositories has emerged as a research direction over the past decade, achieving substantial success in both research and practice to support various software maintenance tasks. Software repositories include bug repository, communication archives, source control repository, etc. When using these repositories to support software maintenance, inclusion of irrelevant information in each repository can lead to decreased effectiveness or even wrong results. ObjectiveThis article aims at selecting the relevant information from each of the repositories to improve effectiveness of software maintenance tasks. MethodFor a maintenance task at hand, maintainers need to implement the maintenance request on the current system. In this article, we propose an approach, MSR4SM, to extract the relevant information from each software repository based on the maintenance request and the current system. That is, if the information in a software repository is relevant to either the maintenance request or the current system, this information should be included to perform the current maintenance task. MSR4SM uses the topic model to extract the topics from these software repositories. Then, relevant information in each software repository is extracted based on the topics. ResultsMSR4SM is evaluated for two software maintenance tasks, feature location and change impact analysis, which are based on four subject systems, namely jEdit, ArgoUML, Rhino and KOffice. The empirical results show that the effectiveness of traditional software repositories based maintenance tasks can be greatly improved by MSR4SM. ConclusionsThere is a lot of irrelevant information in software repositories. Before we use them to implement a maintenance task at hand, we need to preprocess them. Then, the effectiveness of the software maintenance tasks can be improved.

Guest editorial: special section on mining software repositories

Mining software repositories is an increasingly popular and important area of software engineering research aimed at retrieving, integrating, and analyzing data available in various kinds of software repositories—such as version control systems, issue trackers, mailing lists, and user reviews, —to distill useful information about software projects and, whenever possible, to provide recommendations to software engineers. This area of research has been especially active in the last 10-15 years, and researchers have contributed to this area (i) by developing techniques and tools to mine software repositories, (ii) conducting various kinds of evolutionary studies on open source and industrial projects, and (iii) proposing approaches and tools that exploit data from software repositories to provide actionable support to software engineers. Noticeably, the large amount of data available in software repositories—as well as of techniques developed to mine such data—has allowed researchers to perform a lot of empirical research. Three noticeable examples of empirical research in the area of mining software repositories are reported in this special section. In the article “Towards Improving Statistical Modelling of Software Engineering Data: Think Locally, Act Globally!”, the authors study and discuss pros and cons of local defect prediction models—i.e., of models built on subsets of homogeneous data—and global defect prediction models, built on the entire available dataset. Also, they show how it is possible to build a hybrid approach combining local and global models together. The article “Understanding the Impact of Rapid Releases on Software Quality The Case of Firefox” investigates how quality assurance in software projects is impacted when applying rapid release cycles. Specifically, the authors study projects with regular and rapid

A Content Assist based Approach for Providing Rationale of Method Change for Object Oriented Programming

Software engineering requires modification of code during development and maintenance phase. During modification, a difficult task is to understand rationale of changed code. Present Integrated Development Environments (IDEs) attempt to help this by providing features integrated with different types of repositories. However, these features still consume developer's time as he has to switch from editor to another window for this purpose. Moreover, these features focus on elements available in present version of code, thus increasing the difficulty of finding rationale of an element removed or modified earlier. Leveraging different sources for providing information through code completion menus has been shown to be valuable, even when compared to standalone counterparts offering similar functionalities in literature. Literature also shows that it is one of the most used features for consuming information within IDE. Based on that, we prepare an Eclipse plug-in and a framework that allows providing reason of code change, at method granularity, across versions through a new code completion menu in IDE. These allow a software engineer to gain insight about rationale of removed or modified methods which are otherwise not available in present version of code. Professional software engineers participated in our empirical evaluation process and we observed that more than 80% participants considered this to be a useful approach for saving time and effort to understand rationale of method change. Later, based on their feedback, the plug-in and framework is modified to incorporate chronological factors. We perform quasi experimental evaluation with professional software engineers. It is found that time required to find rationale of method change is reduced to at least half compared to usual amount of time required for all the software engineers who participated in the quantitative evaluation. Index Terms—Software engineering, Mining software repositories, experimental software engineering.

Software Dependency Estimation in the code Repositories for the Requirement Evolution

Dependency is the only means to ensure that the source code of a system is consistent with its requirements. During software maintenance and evolution, requirement dependency links become obsolete because dependency model is been not trained properly to updating them. Yet, recovering these dependency links later is a daunting and costly task for building the model for unsupervised enhancements. Consequently, the literature has proposed methods, techniques, and tools to recover these dependency links semi-automatically or automatically. Among the proposed techniques, the literature showed that information retrieval (IR) techniques can automatically recover traceability links between free-text requirements and source code through classification techniques to the Software repositories. However, IR techniques lack accuracy (precision and recall) in terms of Text and concept based mining also leads to code sense disambiguation. In this paper, we show that Semantic mining of software repositories and combining mined results with IR can improve the accuracy (precision and recall) of IR techniques. We apply Dependency Estimation on to compare the accuracy of its dependency links with those recovered using state-of-the-art IR techniques from Vector Space model and Concept based mining. We thus show that mining software repositories and combining the mined data with existing results from IR techniques improves the precision and recall of requirement dependency links.

개발자 별 버그 해결 유형을 고려한 자동적 개발자 추천 접근법

ABSTRACT During the development of the software, a variety of bugs are reported. Several bug tracking systems, such as, Bugzilla, MantisBT, Trac, JIRA, are used to deal with reported bug information in many open source development projects. Bug reports in bug tracking system would be triaged to manage bugs and determine developer who is responsible for resolving the bug report. As the size of the software is increasingly growing and bug reports tend to be duplicated, bug triage becomes more and more complex and difficult. In this paper, we present an approach to assign bug reports to appropriate developers, which is a main part of bug triage task. At first, words which have been included the resolved bug reports are classified according to each developer. Second, words in newly bug reports are selected. After first and second steps, vectors whose items are the selected words are generated. At the third step, TF-IDF(Term frequency - Inverse document frequency) of the each selected words are computed, which is the weight value of each vector item. Finally, the developers are recommended based on the similarity between the developer’s word vector and the vector of new bug report. We conducted an experiment on Eclipse JDT and CDT project to show the applicability of the proposed approach. We also compared the proposed approach with an existing study which is based on machine learning. The experimental results show that the proposed approach is superior to existing method.Keywords:Bug Tracking System, Software Repository Mining, Bug M anagement, Bug Classification, Natural Language Processing

A selection of distinguished papers from the 19th Working Conference on Reverse Engineering 2012

ABSTRACTThis special issue contains a selection of distinguished papers from the 19th Working Conference on Reverse Engineering, held in Kingston, Ontario, Canada on October 15th–18th, 2012. The papers address a variety of topics related to reverse engineering, namely the analysis of execution traces, development of fast and efficient forward static slicing techniques, and mining of software repositories aiming at deriving novel patterns related to reverse engineering activities, such as the migration in Java libraries and the analysis of effects of file editing patterns on source code quality. Copyright © 2014 John Wiley & Sons, Ltd.

다형의 버그 추적 시스템 마이닝 및 분석을 위한 저장소 독립 모델 설계

본 논문은 다양한 버그 추적 시스템으로부터 추출한 데이터를 통합하여 단일 저장소 모델을 제공하는 UniBAS(Unified Bug Analysis System)를 제안한다. UniBAS는 MSR(Mining Software Repositories) 연구 과정에서의 저장소 추출, 데이터 가공이나 모델 생성과 같은 공통적인 반복 작업을 줄이고, 관련 연구자가 상위 수준의 연구에 보다 집중할 수 있도록 함으로써 해당 연구 수행에 발생하는 복잡도와 비용을 줄여준다. 또한, UniBAS는 데이터 추출 뿐 아니라 질의 기반 분석에 필요한 테이블, 뷰 및 저장 프로시저 등을 자동 생성하며, 수집한 데이터 관리와 외부 도구와의 연동을 위해 다양한 형식의 파일을 생성할 수 있다. 사례 연구로 UniBAS의 유용성을 검증하기 위해 Mozilla사이트의 Firefox프로젝트를 대상으로 실제 중복 버그 리포트를 탐지하는 실험을 진행하였다. 이 과정에서 자동 추출된 자료를 대상으로 질의와 분석이 유연하게 이루어질 수 있었으며, 다양한 자연어 처리 알고리즘 적용을 통해 유효한 실험 결과를 얻을 수 있었다. In this paper, we propose UniBAS(Unified Bug Analysis System) to provide a unified repository model by integrating the extracted data from the heterogeneous bug tracking systems. The UniBAS reduces the cost and complexity of the MSR(Mining Software Repositories) research process and enables the researchers to focus on their logics rather than the tedious and repeated works such as extracting repositories, processing data and building analysis models. Additionally, the system not only extracts the data but also automatically generates database tables, views and stored procedures which are required for the researchers to perform query-based analysis easily. It can also generate various types of exported files for utilizing external analysis tools or managing research data. A case study of detecting duplicate bug reports from the Firfox project of the Mozilla site has been performed based on the UniBAS in order to evaluate the usefulness of the system. The results of the experiments with various algorithms of natural language processing and flexible querying to the automatically extracted data also showed the effectiveness of the proposed system.

Automated classification of software change messages by semi-supervised Latent Dirichlet Allocation

ContextTopic models such as probabilistic Latent Semantic Analysis (pLSA) and Latent Dirichlet Allocation (LDA) have demonstrated success in mining software repository tasks. Understanding software change messages described by the unstructured nature-language text is one of the fundamental challenges in mining these messages in repositories. ObjectiveWe seek to present a novel automatic change message classification method characterized by semi-supervised topic semantic analysis. MethodIn this work, we present a semi-supervised LDA based approach to automatically classify change messages. We use domain knowledge of software changes to make labeled samples which are added to build the semi-supervised LDA model. Next, we verify the cross-project analysis application of our method on three open-source projects. Our method has two advantages over existing software change classification methods: First of all, it mitigates the issue of how to set the appropriate number of latent topics. We do not have to choose the number of latent topics in our method, because it corresponds to the number of class labels. Second, this approach utilizes the information provided by the label samples in the training set. ResultsOur method automatically classified about 85% of the change messages in our experiment and our validation survey showed that 70.56% of the time our automatic classification results were in agreement with developer opinions. ConclusionOur approach automatically classifies most of the change messages which record the cause of the software change and the method is applicable to cross-project analysis of software change messages.

Identifying Designers and their Design Knowledge

This paper presents an approach to identify designers who participated in shaping the design of software systems. Designers are identified as well as their design knowledge. Identified designers are recommended to handle a high level change request. High level requests require changing the design of software systems. The identified design knowledge of designers is used to find the appropriate designer to handle a change request. Designers are developers who made design changes to the software. Identifying designers and their design knowledge is based on mining software repositories to extract design changes from commits. Developers who committed any design change are considered designers. Type and amount of committed design changes are used to identify the knowledge of designers. A tool has been developed to automatically recommend a set of designers to handle a high level change request. Recommended designers are ranked based on their knowledge that is identified from cumulative design activities for designers. The tool also extracts useful information and statistics about designers and their activities. A case study of two open source projects is presented to show the benefits of the proposed approach and tool. Results showed that most committers are not designers. It also shows that designers can be ranked based on their design knowledge.

Special issue: a selection of distinguished papers from the 18th Working Conference on Reverse Engineering 2011

Special issue: a selection of distinguished papers from the 18th Working Conference on Reverse Engineering 2011

Declarative visitors to ease fine-grained source code mining with full history on billions of AST nodes

Software repositories contain a vast wealth of information about software development. Mining these repositories has proven useful for detecting patterns in software development, testing hypotheses for new software engineering approaches, etc. Specifically, mining source code has yielded significant insights into software development artifacts and processes. Unfortunately, mining source code at a large-scale remains a difficult task. Previous approaches had to either limit the scope of the projects studied, limit the scope of the mining task to be more coarse-grained, or sacrifice studying the history of the code due to both human and computational scalability issues. In this paper we address the substantial challenges of mining source code: a) at a very large scale; b) at a fine-grained level of detail; and c) with full history information. To address these challenges, we present domain-specific language features for source code mining. Our language features are inspired by object-oriented visitors and provide a default depth-first traversal strategy along with two expressions for defining custom traversals. We provide an implementation of these features in the Boa infrastructure for software repository mining and describe a code generation strategy into Java code. To show the usability of our domain-specific language features, we reproduced over 40 source code mining tasks from two large-scale previous studies in just 2 person-weeks. The resulting code for these tasks show between 2.0x--4.8x reduction in code size. Finally we perform a small controlled experiment to gain insights into how easily mining tasks written using our language features can be understood, with no prior training. We show a substantial number of tasks (77%) were understood by study participants, in about 3 minutes per task.

Detecting asynchrony and dephase change patterns by mining software repositories

SUMMARYSoftware maintenance accounts for the largest part of the costs of any program. During maintenance activities, developers implement changes (sometimes simultaneously) on artifacts in order to fix bugs and to implement new requirements. To reduce this part of the costs, previous work proposed approaches to identify the artifacts of programs that change together. These approaches analyze historical data, mined from version control systems, and report change patterns, which lead at the causes, consequences, and actors of the changes to source code files. They also introduce so‐called change patterns that describe some typical change dependencies among files. In this paper, we introduce two novel change patterns: the asynchrony change pattern, corresponding to macro co‐changes (MC), that is, of files that co‐change within a large time interval (change periods) and the dephase change pattern, corresponding to dephase macro co‐changes (DC), that is, MC that always happens with the same shifts in time. We present our approach, that we named Macocha, to identify these two change patterns in large programs. We use the k‐nearest neighbor algorithm to group changes into change periods. We also use the Hamming distance to detect approximate occurrences of MC and DC. We apply Macocha and compare its performance in terms of precision and recall with UMLDiff (file stability) and association rules (co‐changing files) on seven systems: ArgoUML, FreeBSD, JFreeChart, Openser, SIP, XalanC, and XercesC developed with three different languages (C, C++, and Java). These systems have a size ranging from 532 to 1693 files, and during the study period, they have undergone 1555 to 23,944 change commits. We use external information and static analysis to validate (approximate) MC and DC found by Macocha. Through our case study, we show the existence and usefulness of these novel change patterns to ease software maintenance and, potentially, reduce related costs. Copyright © 2013 John Wiley & Sons, Ltd.

A framework for acquisition and application of software architecture evolution knowledge

Software systems continuously evolve as a consequence of frequent changes in their functional requirements and the environment surrounding them. Architecture-centric software evolution (ACSE) enables changes in software structure and behaviour while abstracting the complex implementation-specific details. However, due to recurring evolution there is a need for solutions that enable a systematic reuse of frequent changes in software architectures. In recent years, architecture change patterns and evolution styles proved successful in promoting reuse expertise to tackle architecture evolution. However, there do not exist any solutions that enable a continuous acquisition and application of architecture evolution knowledge to systematically address frequent changes in software architectures. In this paper, we propose a framework PatEvol that aims to unify the concepts of i) software repository mining and ii) software evolution to enable acquisition and application of architecture evolution knowledge. In the proposed PatEvol framework, we present knowledge acquisition ( architecture evolution mining ) to enable post-mortem analysis of evolution histories to empirically discover evolution-centric knowledge. Furthermore, we support reuse of discovered knowledge to enable knowledge application ( architecture evolution execution ) that enables evolution-off-the-shelf in software architectures. Tool support facilitates the knowledge acquisition and knowledge application processes in the PatEvol framework.

Introduction to the special issue on mining software repositories

The Mining Software Repositories field analyzes therich data available insoftwarerepositories to uncover interesting and actionable information about software systems and projects. Thanks to the ready availability of software configuration management, mailing list, and bug tracking repositories from open source projects, it has gained popularity since 2004 with the first instance of theMSR workshop (nowconference) andcontinues tobeoneof thefastestgrowing fields in the area of software engineering. Researchers in this field empirically explore a range of software engineering questions using software repository data as the primary source of information. Some commonly explored areas include software evolution, models of software development processes, characterization of developers and their activities, prediction of future software qualities, use of machine learning techniques on software project data, software bug prediction, analysis of software change patterns, and analysis of code clones. There has also been a stream of work on tools for mining software repositories, and techniques for visualizing software repository data. In recent years the importance of the field has further increasedas today's society andbusinesses become more data-driven. Industry has a strong interest in transforming software repository data into actionable insights to inform better development decisions. Analytics is already commonly used in many businesses—notably in marketing, to better reach and understand customers (May 2009). The application of analytics to software development is becoming more popular. For those wishing to develop a broad understanding of the software repository mining research field, there are several resources.

Trustrace: Mining Software Repositories to Improve the Accuracy of Requirement Traceability Links

Traceability is the only means to ensure that the source code of a system is consistent with its requirements and that all and only the specified requirements have been implemented by developers. During software maintenance and evolution, requirement traceability links become obsolete because developers do not/cannot devote effort to updating them. Yet, recovering these traceability links later is a daunting and costly task for developers. Consequently, the literature has proposed methods, techniques, and tools to recover these traceability links semi-automatically or automatically. Among the proposed techniques, the literature showed that information retrieval (IR) techniques can automatically recover traceability links between free-text requirements and source code. However, IR techniques lack accuracy (precision and recall). In this paper, we show that mining software repositories and combining mined results with IR techniques can improve the accuracy (precision and recall) of IR techniques and we propose Trustrace, a trust--based traceability recovery approach. We apply Trustrace on four medium-size open-source systems to compare the accuracy of its traceability links with those recovered using state-of-the-art IR techniques from the literature, based on the Vector Space Model and Jensen-Shannon model. The results of Trustrace are up to 22.7 percent more precise and have 7.66 percent better recall values than those of the other techniques, on average. We thus show that mining software repositories and combining the mined data with existing results from IR techniques improves the precision and recall of requirement traceability links.

Report on the fourth workshop on hot topics in software upgrades (HotSWUp 2012)

The Fourth Workshop on Hot Topics in Software Upgrades (HotSWUp 2012) was held on June 3, 2012 in Zurich, Switzerland. The workshop was co-located with ICSE 2012. The goal of HotSWUp is to identify, through interdisciplinary collaboration, cutting-edge research ideas for implementing software upgrades. The workshop combined presentations of peer-reviewed research papers with a keynote speech on how empirical software engineering can help reduce update-induced failures. The audience included researchers and practitioners from academia and industry. In addition to the technical presentations, the program allowed ample time for discussions, which were driven by debate questions provided in advance by the presenters. HotSWUp provides a premier forum for discussing problems that are often considered niche topics in the established research communities. For example, the technical discussions at HotSWUp'12 covered dynamic software updates, package management tools, using model-checking and verification to verify updates, empirical software engineering and repository mining, and highlighted many synergies among these and other topics.

OSSGrab: Software Repositories and App Store Mining Tool

One of the main challenges for empirical researchers is to collect software data. However, with the emergence of the open source repositories, they have a large amount of software to choose from for mostly any types of research. Moreover, the mining software repositories research area can be further extended to the mobile apps mining. Generally, searching through large software repositories to look for specific systems can be a daunting task. Therefore, there is a need to build a tool which can expedite and ease the search process so that the researchers can focus on analyzing the data. Our paper presents a tool, OSSGrab, which can be used to automate the search process in the SourceForge software repository, as well as searching through the Android app store. As a result, this tool has managed to save tremendous amount of time that need to be spent for data collection.

Integrating Software Repository Mining: A Decision Support Centered Approach

Mining software repositories (MSR) research had significantly contributed to software engineering. However, MSR results integration across repo sitories is a recent concern that is getting more attention from researchers each day. Some noticeable research in this sense is related to the approximation between MSR and semantic web, specially linked data approaches which makes it possible to integrat e repositories and mined results. Manifested that way, we believe that current research is not fully addressing the practical integration of MSR results, specially, in software engineering due to not considering that these results needs to be integrated tothe tools as assistance to activity performers, as a kind of decision making support. Based on this statement this research describes an approach, named Sambasore, which is concerned with MSR results inter -repository integration and also to decision makin g support processes, based on tool assistance modelling. To show its feasibility we describe the main concepts, some related works and also a proof of concept experiment applied to a software process modelling tool named Spider PM.