CCMC: Code Completion with a Memory Mechanism and a Copy Mechanism

Abstract

Code completion tools are increasingly important when developing modern software. Recently, statistical language modeling techniques have achieved great success in the code completion task. However, two major issues with these techniques severely affect the performance of neural language models (NLMs) of code completion. a) Long-range dependences are common in program source code. b) New and rare vocabulary in code is much higher than natural language. To address the challenges above, in this paper, we propose code completion with a memory mechanism and a copy mechanism (CCMC). To capture the long-range dependencies in the program source code, we employ Transformer-XL as our base model. To utilize the locally repeated terms in program source code, we apply the pointer network into our base model and design CopyMask to improve the training efficiency, which is inspired by masked multihead attention in the transformer decoder. To combine the long-range dependency modeling ability from Transformer-XL and the ability to copy the input token to output from the pointer network, we design a memory mechanism and a copy mechanism. Through our memory mechanism, our model can uniformly manage the context used by Transformer-XL and pointer network. Through our copy mechanism, our model can either generate a within-vocabulary token or copy an out-of-vocabulary (OOV) token from inputs. Experiments on a real-world dataset demonstrate the effectiveness of our CCMC on the code completion task.