Cross-domain arrhythmia classification (CAC) aims to transfer the model trained on a label-sufficient source domain to a label-scarce target domain. To the best of our knowledge, almost all existing CAC models focus on the unsupervised setting, where no labeled target samples are available. However, in most practical scenes, acquiring limited annotated target samples is feasible, which can provide reliable target semantic information for model learning directly. Consequently, we first propose a more realistic semi-supervised CAC setting, where only the source samples and extremely limited target samples are annotated. Most previous CAC models realize cross-domain learning by aligning the feature distributions of source and target domains coarsely and globally, where the semantic invariance within each class is not taken into consideration during the domain alignment process. Additionally, the semantic information contained in the feature space is not fully utilized for target pseudo label mining. To address the above two issues, a unified framework containing the semantic-aware feature alignment (SAFA) and prototype-based label propagation (PBLP) modules is proposed. In the proposed framework, SAFA and PBLP are complementary to each other. Specifically, SAFA provides more robust prototypes for PBLP by performing semantic-aware feature alignment, and PBLP offers more reliable target pseudo labels for more effective semantic-aware feature alignment learning. Comprehensive qualitative and quantitative experimental results on different benchmarks verify the effectiveness of the proposed method.