We present a generic and accurate method to transfer part-level labels from annotated 3D shapes to images or annotated images to 3D shapes. The label transfer problem is formulated as a Conditional random field (CRF) model, in which a novel local correspondence encoding term encoding the probability of label assignment of each pixel as a label histogram serves as the likelihood of the labels. With the aid of local correspondence encoding, highly accurate part-level label transfer results can be easily achieved. Our method works well for the bi-directional label transfer between 3D shapes and images, suitable to image-guided 3D shape segmentation, 3D shape-guided image segmentation, and has also found various applications. We thoroughly evaluate our method and demonstrate its superiority over the state-of-the-art methods through experiments on benchmark datasets.