The efficiency in breast cancer from imaging-based computer-aided diagnosis (CAD) has been revealed in recent years. As a fact, the methods grounded on a single modality constantly lack behind multimodal CAD imaging. However, owing to the restrictions of imaging devices, expressly in rural hospitals, single-modal imaging becomes a favorite in clinical practice for diagnosis. A fresh learning model trending nowadays known as learning using privileged information (LUPI) adopts additional privileged information (PI) modality to help during the training stage, but PI does not contribute in the testing stage. Meanwhile, the link exists between PI and training samples; the same is then reassigned to the learned model. We propose a LUPI-based CAD framework for breast cancer using privileged information in this work. The work offers both a classifier- or feature-level LUPI, in which the information is shifted from the additional PI modality to the diagnosis modality. A thorough comparison has been made among six classifier-level algorithms and six feature-level LUPI algorithms. The experimental results on both the acquired primary datasets show that all classifier-level and deep learning-based feature-level LUPI algorithms can enhance the performance of a single-modal imaging-based CAD for breast cancer by relocating PI.