The fault diagnosis of chillers is of great significance in reducing building energy consumption and extending the operational lifespan of refrigeration equipment. Several popular machine learning-based fault diagnosis methods rely heavily on many labeled samples. However, such samples are difficult to obtain in practice due to sparse fault data and high labeling costs. This limits the application of ML-based fault diagnosis methods based on supervised learning. To reduce the dependence on labeled samples, this paper proposes a novel chiller fault diagnosis method based on neighbor-optimized graph convolutional network. The method improves the utilization efficiency of unlabeled samples by mining the spatio-temporal relationship between a large number of unlabeled samples and a limited number of labeled samples. And it dynamically adjusts the graph's structure by optimizing the number of adjacent samples in the correlation graph to obtain better diagnostic results. Its effectiveness is validated on the authoritative dataset ASHRAE RP-1043 and a more challenging dataset of real-world chillers in a building. Experimental results show that the proposed method can achieve better diagnostic performance than the state-of-the-art methods.