Advanced artificial intelligence techniques, such as federated learning, has been applied to broad areas, e.g., image classification, speech recognition, smart city, and healthcare. Despite intensive research on federated learning, existing schemes are vulnerable to attacks and can hardly meet the security requirements for real-world applications. The problem of designing a secure federated learning framework to ensure the correctness of training procedure has not been sufficiently studied and remains open. In this paper, we propose VFChain, a verifiable and auditable federated learning framework based on the blockchain system. First, to provide the verifiability, a committee is selected through the blockchain to collectively aggregate models and record verifiable proofs in the blockchain. Then, to provide the auditability, a novel authenticated data structure is proposed for blockchain to improve the search efficiency of verifiable proofs and support a secure rotation of committee. Finally, to further improve the search efficiency, an optimization scheme is proposed to support multiple-model learning tasks. We implement VFChain and conduct extensive experiments by utilizing the popular deep learning models over the public real-world dataset. The evaluation results demonstrate the effectiveness of our proposed VFChain system.