Program Comprehension Activities Research Articles

Multi-Intent Inline Code Comment Generation via Large Language Model

Code comment generation typically refers to the process of generating concise natural language descriptions for a piece of code, which facilitates program comprehension activities. Inline code comments, as a part of code comments, are also crucial for program comprehension. Recently, the emergence of large language models (LLMs) has significantly boosted the performance of natural language processing tasks. This naturally inspires us to explore the performance of the LLMs in the task of inline code comment generation. To this end, we evaluate open-source LLMs on a large-scale dataset and compare the results with the current state-of-the-art methods. Specifically, we explore the model performance in the following scenarios based on the widely used evaluation metrics (i.e. BLEU, Meteor, and ROUGE-L): (1) generation with simple instruction; (2) few-shot-guided generation with random examples selected from the database; (3) few-shot-guided generation with similar examples selected from the database; and (4) adopt the re-ranking strategy for the output of LLMs. Our findings reveal that: (1) under the simple instruction scenario, LLMs could not fully show the potential in the task of inline comment generation compared to the state-of-the-art models; (2) random few-shot leads to a slight improvement; (3) similar few-shot and re-ranking strategy could significantly enhance the performance of LLMs; and (4) for inline comment and code snippet pairs with different intents, why category achieves the best performance and what category achieves relatively poorer performance. That remains consistent across all four scenarios. Our findings shed light on future research directions for using LLMs in inline comment generation tasks.

What do class comments tell us? An investigation of comment evolution and practices in Pharo Smalltalk

ContextPrevious studies have characterized code comments in various programming languages, showing how high quality of code comments is crucial to support program comprehension activities, and to improve the effectiveness of maintenance tasks. However, very few studies have focused on understanding developer practices to write comments. None of them has compared such developer practices to the standard comment guidelines to study the extent to which developers follow the guidelines.ObjectiveTherefore, our goal is to investigate developer commenting practices and compare them to the comment guidelines.MethodThis paper reports the first empirical study investigating commenting practices in Pharo Smalltalk. First, we analyze class comment evolution over seven Pharo versions. Then, we quantitatively and qualitatively investigate the information types embedded in class comments. Finally, we study the adherence of developer commenting practices to the official class comment template over Pharo versions.ResultsOur results show that there is a rapid increase in class comments in the initial three Pharo versions, while in subsequent versions developers added comments to both new and old classes, thus maintaining a similar code to comment ratio. We furthermore found three times as many information types in class comments as those suggested by the template. However, the information types suggested by the template tend to be present more often than other types of information. Additionally, we find that a substantial proportion of comments follow the writing style of the template in writing these information types, but they are written and formatted in a non-uniform way.ConclusionThe results suggest the need to standardize the commenting guidelines for formatting the text, and to provide headers for the different information types to ensure a consistent style and to identify the information easily. Given the importance of high-quality code comments, we draw numerous implications for developers and researchers to improve the support for comment quality assessment tools.

How to identify class comment types? A multi-language approach for class comment classification

Most software maintenance and evolution tasks require developers to understand the source code of their software systems. Software developers usually inspect class comments to gain knowledge about program behavior, regardless of the programming language they are using. Unfortunately, (i) different programming languages present language-specific code commenting notations and guidelines; and (ii) the source code of software projects often lacks comments that adequately describe the class behavior, which complicates program comprehension and evolution activities. To handle these challenges, this paper investigates the different language-specific class commenting practices of three programming languages: Python, Java, and Smalltalk. In particular, we systematically analyze the similarities and differences of the information types found in class comments of projects developed in these languages. We propose an approach that leverages two techniques –namely Natural Language Processing and Text Analysis –to automatically identify class comment types, i.e., the specific types of semantic information found in class comments. To the best of our knowledge, no previous work has provided a comprehensive taxonomy of class comment types for these three programming languages with the help of a common automated approach. Our results confirm that our approach can classify frequent class comment information types with high accuracy for the Python, Java, and Smalltalk programming languages. We believe this work can help in monitoring and assessing the quality and evolution of code comments in different programming languages, and thus support maintenance and evolution tasks.

Automatic and Accurate Expansion of Abbreviations in Parameters

Abbreviations are widely used in identifiers. However, they have severe negative impact on program comprehension and IR-based software maintenance activities, e.g., concept location, software clustering, and recovery of traceability links. Consequently, a number of efficient approaches have been proposed successfully to expand abbreviations in identifiers. Most of such approaches rely heavily on dictionaries, and rarely exploit the specific and fine-grained context of identifiers. As a result, such approaches are less accurate in expanding abbreviations (especially short ones) that may match multiple dictionary words. To this end, in this paper we propose an automatic approach to improve the accuracy of abbreviation expansion by exploiting the specific and fine-grained context. It focuses on a special but common category of abbreviations (abbreviations in parameter names), and thus it can exploit the specific and fine-grained context, i.e., the type of the enclosing parameter as well the corresponding formal (or actual) parameter name. The recent empirical study on parameters suggest that actual parameters are often lexically similar to their corresponding formal parameters. Consequently, it is likely that an abbreviation in a formal parameter can find its full terms in the corresponding actual parameter, and vice versa. Based on this assumption, a series of heuristics are proposed to look for full terms from the corresponding actual (or formal) parameter names. To the best of our knowledge, we are the first to expand abbreviations by exploiting the lexical similarity between actual and formal parameters. We also search for full terms in the data type of the enclosing parameter. Only if all such heuristics fail, the approach turns to the traditional abbreviation dictionaries. We evaluate the proposed approach on seven well known open-source projects. Evaluation results suggest that when only parameter abbreviations are involved, the proposed approach can improve the precision from 26 to 95 percent and recall from 26 to 65 percent compared against the state-of-the-art general purpose approach. Consequently, the proposed approach could be employed as a useful supplement to existing approaches to expand parameter abbreviations.

Finding key classes in object-oriented software systems by techniques based on static analysis

Abstract Context Software maintenance is burdened by program comprehension activities which consume a big part of project resources. Program comprehension is difficult because the code to be analyzed is very large and the documentation may not be well structured to help navigating through the code. Objective Tools should support the early stages of program comprehension. Our goal is to build tools that analyze the code and filter this large amount of information such that only the most important information is presented to the software maintenance team. In the case of object-oriented systems, finding the important information means finding the most important classes, also called the key classes of the system. Method In this work, we formulate and explore several hypotheses regarding which are the class attributes that characterize important classes. By class attributes, we understand here different metrics that quantify properties of the class such as its connections and relationships with other classes. All the necessary input data for computing class attributes are extracted from code by static analysis. We experimentally investigate which attributes are best suited to rank classes according to their importance, doing an extensive empirical study on fifteen software systems. Result Attributes from the categories of direct connections and network centrality are the best for finding key classes. We identified three class attributes which are best as class ranking criteria: PR-U2-W and CONN-TOTAL-W when the target set of key classes is small and CONN-TOTAL when the target set has a large and variable size. We show that the method of ranking classes based on these attributes outperforms known related work approaches of finding key classes. Conclusions Our method allows us to build easy-to-use fully automatic tools which find almost instantly the key classes of a software system starting from its code.

Measuring Program Comprehension: A Large-Scale Field Study with Professionals

During software development and maintenance, developers spend a considerable amount of time on program comprehension activities. Previous studies show that program comprehension takes up as much as half of a developer’s time. However, most of these studies are performed in a controlled setting, or with a small number of participants, and investigate the program comprehension activities only within the IDEs. However, developers’ program comprehension activities go well beyond their IDE interactions. In this paper, we extend our ActivitySpace framework to collect and analyze Human-Computer Interaction (HCI) data across many applications (not just the IDEs). We follow Minelli et al.’s approach to assign developers’ activities into four categories: navigation, editing, comprehension, and other. We then measure the comprehension time by calculating the time that developers spend on program comprehension, e.g., inspecting console and breakpoints in IDE, or reading and understanding tutorials in web browsers. Using this approach, we can perform a more realistic investigation of program comprehension activities, through a field study of program comprehension in practice across a total of seven real projects, on 78 professional developers, and amounting to 3,148 working hours. Our study leverages interaction data that is collected across many applications by the developers. Our study finds that on average developers spend $\sim$ 58 percent of their time on program comprehension activities, and that they frequently use web browsers and document editors to perform program comprehension activities. We also investigate the impact of programming language, developers’ experience, and project phase on the time that is spent on program comprehension, and we find senior developers spend significantly less percentages of time on program comprehension than junior developers. Our study also highlights the importance of several research directions needed to reduce program comprehension time, e.g., building automatic detection and improvement of low quality code and documentation, construction of software-engineering-specific search engines, designing better IDEs that help developers navigate code and browse information more efficiently, etc.

Alternatives to the Knowledge Discovery Metamodel: An Investigation

To better understand and exploit the knowledge necessary to comprehend and evolve an existing system, different models can be extracted from it. Models represent the extracted information at various abstraction levels, and are useful to document, maintain, and reengineer the system. The Knowledge Discovery Metamodel (KDM) has been defined by the object management group as a meta-model supporting a large share of reverse engineering activities. Its specification has also been adopted by the ISO in 2012. This paper explores and describes alternative meta-models proposed in the literature to support reverse engineering, program comprehension, and software evolution activities. We focus on the similarity and differences of the alternative meta-models with KDM, trying to understand the potentials of reciprocal information interchange. We describe KDM and other five meta-models, plus their extensions available in the literature and their diffusion in the reverse engineering community. We also investigate the approaches using KDM and the five meta-models. In the paper, we underline the limited reuse of models for reverse engineering, and identify potential directions for future related research, to enhance the existing models and ease the exchange of information among them.

An automated approach for noise identification to assist software architecture recovery techniques

Software systems’ concrete architecture often drifts from the intended architecture throughout their evolution. Program comprehension activities, like software architecture recovery, become very demanding, especially for large and complex systems due to the existence of noise, which is created by omnipresent and utility classes that obscure the system structure. Omnipresent classes represent crosscutting concerns, utilities or elementary domain concepts. The identification and filtering of noise is a necessary preprocessing step before attempting program comprehension techniques, especially for undocumented systems. In this paper, we propose an automated methodology for noise identification. Our methodology is based on the notion that noisy classes are widely used in a system, directly or indirectly. We combine classes’ usage significance with their participation in the system’s subgraphs, in order to identify the classes that are persistently used. Usage significance is measured according to Component Rank, a well-established metric in the literature, which ranks software artifacts according to their usage significance. The experimental results show that the proposed methodology successfully captures classes that produce noise and improves the results of existing algorithms for software systems’ architectural decomposition.

Configuring latent Dirichlet allocation based feature location

Feature location is a program comprehension activity, the goal of which is to identify source code entities that implement a functionality. Recent feature location techniques apply text retrieval m...

Identification of behavioural and creational design motifs through dynamic analysis

AbstractDesign patterns offer design motifs, solutions to object‐oriented design problems. Design motifs lead to well‐structured designs and thus are believed to ease software maintenance. However, after use, they are often ‘lost’ and are consequently of little help during program comprehension and other maintenance activities. Therefore, several works proposed design pattern identification approaches to recover occurrences of the motifs. These approaches mainly used the structure and organization of classes as input. Consequently, they have a low precision when considering behavioural and creational motifs, which pertain to the assignment of responsibilities and the collaborations among objects at runtime. We propose MoDeC, an approach to describe behavioural and creational motifs as collaborations among objects in the form of scenario diagrams. We identify these motifs using dynamic analysis and constraint programming. Using a proof‐of‐concept implementation of MoDeC and different scenarios for five other Java programs and Builder, Command, and Visitor, we show that MoDeC has a better precision than the state‐of‐the‐art static approaches. Copyright © 2009 John Wiley & Sons, Ltd.

A controlled experiment in assessing and estimating software maintenance tasks

Context Software maintenance is an important software engineering activity that has been reported to account for the majority of the software total cost. Thus, understanding the factors that influence the cost of software maintenance tasks helps maintainers to make informed decisions about their work. Objective This paper describes a controlled experiment of student programmers performing maintenance tasks on a C++ program. The objective of the study is to assess the maintenance size, effort, and effort distributions of three different maintenance types and to describe estimation models to predict the programmer’s effort spent on maintenance tasks. Method Twenty-three graduate students and a senior majoring in computer science participated in the experiment. Each student was asked to perform maintenance tasks required for one of the three task groups. The impact of different LOC metrics on maintenance effort was also evaluated by fitting the data collected into four estimation models. Results The results indicate that corrective maintenance is much less productive than enhancive and reductive maintenance and program comprehension activities require as much as 50% of the total effort in corrective maintenance. Moreover, the best software effort model can estimate the time of 79% of the programmers with the error of or less than 30%. Conclusion Our study suggests that the LOC added, modified, and deleted metrics are good predictors for estimating the cost of software maintenance. Effort estimation models for maintenance work may use the LOC added, modified, deleted metrics as the independent parameters instead of the simple sum of the three. Another implication is that reducing business rules of the software requires a sizable proportion of the software maintenance effort. Finally, the differences in effort distribution among the maintenance types suggest that assigning maintenance tasks properly is important to effectively and efficiently utilize human resources.