Automatic smart contract comment generation via large language models and in-context learning

Abstract

Context:Designing effective automatic smart contract comment generation approaches can facilitate developers’ comprehension, boosting smart contract development and improving vulnerability detection. The previous approaches can be divided into two categories: fine-tuning paradigm-based approaches and information retrieval-based approaches. Objective:However, for the fine-tuning paradigm-based approaches, the performance may be limited by the quality of the gathered dataset for the downstream task and they may have knowledge-forgetting issues, which can reduce the generality of the fine-tuned model. While for the information retrieval-based approaches, it is difficult for them to generate high-quality comments if similar code does not exist in the historical repository. Therefore we want to utilize the domain knowledge related to smart contract code comment generation in large language models (LLMs) to alleviate the disadvantages of these two types of approaches. Method:In this study, we propose an approach SCCLLM based on LLMs and in-context learning. Specifically, in the demonstration selection phase, SCCLLM retrieves the top-k code snippets from the historical corpus by considering syntax, semantics, and lexical information. In the in-context learning phase, SCCLLM utilizes the retrieved code snippets as demonstrations for in-context learning, which can help to utilize the related knowledge for this task in the LLMs. In the LLMs inference phase, the input is the target smart contract code snippet, and the output is the corresponding comment generated by the LLMs. Results:We select a large corpus from a smart contract community Etherscan.io as our experimental subject. Extensive experimental results show the effectiveness of SCCLLM when compared with baselines in automatic evaluation and human evaluation. We also show the rationality of our customized demonstration selection strategy in SCCLLM by ablation studies. Conclusion:Our study shows using LLMs and in-context learning is a promising direction for automatic smart contract comment generation, which calls for more follow-up studies.