Code Comprehension Tasks Research Articles

Examining Factors Influencing Cognitive Load of Computer Programmers.

In this study, the factors influencing the cognitive load of computer programmers during the perception of different code tasks were investigated. The eye movement features of computer programmers were used to provide a significant relationship between the perceptual processes of the sample codes and cognitive load. Thanks to the relationship, the influence of various personal characteristics of programmers on cognitive load was examined. Various personal parameters such as programming experience, age, native language, and programming frequency were used in the study. The study was performed on the Eye Movements in Programming (EMIP) dataset containing 216 programmers with different characteristics. Eye movement information recorded during two different code comprehension tasks was decomposed into sub-information, such as pupil movement speed and diameter change. Rapid changes in eye movement signals were adaptively detected using the z-score peak detection algorithm. Regarding the cognitive load calculations, canonical correlation analysis was used to build a statistically significant and efficient mathematical model connecting the extracted eye movement features and the different parameters of the programmers, and the results were statistically significant. As a result of the analysis, the factors affecting the cognitive load of computer programmers for the related database were converted into percentages, and it was seen that linguistic distance is an essential factor in the cognitive load of programmers and the effect of gender on cognitive load is quite limited.

SelfCode: An Annotated Corpus and a Model for Automated Assessment of Self-Explanation During Source Code Comprehension

The ability to automatically assess learners' activities is the key to user modeling and personalization in adaptive educational systems.The work presented in this paper opens an opportunity to expand the scope of automated assessment from traditional programming problems to code comprehension tasks where students are requested to explain the critical steps of a program. The ability to automatically assess these self-explanations offers a unique opportunity to understand the current state of student knowledge, recognize possible misconceptions, and provide feedback. Annotated datasets are needed to train Artificial Intelligence/Machine Learning approaches for the automated assessment of student explanations. To answer this need, we present a novel corpus called SelfCode which consists of 1,770 sentence pairs of student and expert self-explanations of Java code examples, along with semantic similarity judgments provided by experts. We also present a baseline automated assessment model that relies on textual features. The corpus is available at the GitHub repository (https://github.com/jeevanchaps/SelfCode).

Beyond Literal Meaning: Uncover and Explain Implicit Knowledge in Code Through Wikipedia-Based Concept Linking

When reusing or modifying code, developers need to understand the implicit knowledge behind a piece of code in addition to the literal meaning of code. Such implicit knowledge involves related concepts and their explanations. Uncovering and understanding the implicit knowledge in code are challenging due to the extensive use of abbreviations, scattered expressions of concepts, and ambiguity of concept mentions. In this paper, we propose an automatic approach (called CoLiCo) that can uncover implicit concepts in code and link the uncovered concepts to Wikipedia. Based on a trained identifier embedding model, CoLiCo identifies Wikipedia concepts mentioned in a given code snippet and excerpts a paragraph-level explanation from Wikipedia for each concept. During the process, CoLiCo resolves identifier abbreviation (i.e., concepts mentioned in the form of abbreviations) and identifier aggregation (i.e., concepts mentioned by an aggregation of multiple identifiers) based on identifier embedding and mining of identifier abbreviation/aggregation relations. Experimental study shows that CoLiCo outperforms a general entity linking approach by 38.7% in the correctness of concept linking and identifies 96.7% more correct concept linkings on a dataset with 629 code snippets. The concept linking is significant for program understanding in 54% code snippets. Our user study shows that CoLiCo can significantly shorten the time and improve the correctness in code comprehension tasks that intensively involve implicit knowledge.

An empirical evaluation of machine learning techniques to classify code comprehension based on EEG data

Context:Code comprehension consists of a cognitive process in which developers invest mental effort to grasp source code snippets. The literature evidence that electroencephalogram (EEG) signals correlate with developers’ code comprehension. An effective machine learning technique that classifies code comprehension based on EEG would be a means for recommendation systems to suggest an appropriate task for the developer. Problem:However, there is a lack of empirical studies to investigate the effectiveness of machine learning techniques for classifying developers’ code comprehension based on their EEG data. Objective:This study conducts an empirical study to analyze the effectiveness of machine learning techniques to classify developers’ code comprehension trained with EEG Data. Method:A wireless EEG device was used to collect brainwaves from software developers while performing code comprehension tasks. The generated data set was used to train a K-Nearest Neighbor (KNN), Neural Network (NN), a Naïve Bayes (NB), a Random Forest (RF), and a Support Vector Machine (SVM) classifier. We measured the effectiveness of these techniques using precision, recall, and f-measure. Furthermore, we analyzed the mean differences of f-measure between classifiers to check if the classifiers’ effectiveness is significantly higher than the f-measure mean of a random guessing classifier and higher than a high effectiveness threshold of 80%. Results:After we trained the classifiers using 10-fold cross-validation, the K-Nearest Neighbor classifier obtained an f-measure mean of 86% to classify code comprehension. Moreover, the results also show that KNN, NN, and RF techniques were able to classify code comprehension based on developers’ EEG data higher with an f-measure above 80%.

EMIP: The eye movements in programming dataset

A large dataset that contains the eye movements of N=216 programmers of different experience levels captured during two code comprehension tasks is presented. Data are grouped in terms of programming expertise (from none to high) and other demographic descriptors. Data were collected through an international collaborative effort that involved eleven research teams across eight countries on four continents. The same eye tracking apparatus and software was used for the data collection. The Eye Movements in Programming (EMIP) dataset is freely available for download. The varied metadata in the EMIP dataset provides fertile ground for the analysis of gaze behavior and may be used to make novel insights about code comprehension.

RePOR: Mimicking humans on refactoring tasks. Are we there yet?

Refactoring is a maintenance activity that aims to improve design quality while preserving the behavior of a system. Several (semi)automated approaches have been proposed to support developers in this maintenance activity, based on the correction of anti-patterns, which are “poor” solutions to recurring design problems. However, little quantitative evidence exists about the impact of automatically refactored code on program comprehension, and in which context automated refactoring can be as effective as manual refactoring. Leveraging RePOR, an automated refactoring approach based on partial order reduction techniques, we performed an empirical study to investigate whether automated refactoring code structure affects the understandability of systems during comprehension tasks. (1) We surveyed 80 developers, asking them to identify from a set of 20 refactoring changes if they were generated by developers or by a tool, and to rate the refactoring changes according to their design quality; (2) we asked 30 developers to complete code comprehension tasks on 10 systems that were refactored by either a freelancer or an automated refactoring tool. To make comparison fair, for a subset of refactoring actions that introduce new code entities, only synthetic identifiers were presented to practitioners. We measured developers’ performance using the NASA task load index for their effort, the time that they spent performing the tasks, and their percentages of correct answers. Our findings, despite current technology limitations, show that it is reasonable to expect a refactoring tools to match developer code. Indeed, results show that for 3 out of the 5 anti-pattern types studied, developers could not recognize the origin of the refactoring (i.e., whether it was performed by a human or an automatic tool). We also observed that developers do not prefer human refactorings over automated refactorings, except when refactoring Blob classes; and that there is no statistically significant difference between the impact on code understandability of human refactorings and automated refactorings. We conclude that automated refactorings can be as effective as manual refactorings. However, for complex anti-patterns types like the Blob, the perceived quality achieved by developers is slightly higher.

Software code complexity assessment using EEG features.

This paper provides a study using Electroencephalography (EEG) to investigate the brain activity during code comprehension tasks. Three different code complexity levels according to five complexity metrics were considered. The use of EEG for this purpose is relevant, since the existing studies were mostly focused on neuroimaging techniques. Using Leave-One-Subject-Out cross-validation procedure for 30 subjects, it was found that the features related with the Gamma activity were the most common in all the folds. Regarding the brain regions, right parietal was the most frequent region contributing with more features. A Linear Discriminant Analysis Classifier for task classification, obtained a F-Measure of 92.71% for Code complexity easy, 52.25% for Code complexity intermediate and 53.13% for Code complexity advanced, revealing an evidence of mental effort saturation with the code complexity degree. This suggests that current code complexity metrics do not capture cognitive load and might not be the best approach to assess bug risk.