Nowadays, leveraging data augmentation-based methods to address the data-shortage problem in diagnosis field becomes fairly prevailing, while assessing the quality of the generated data receives little attention. Typically, the data quality is evaluated by straightforward employing some existing shallow functions or simple classification models, which have several disadvantages. In this paper, the limitations of existing techniques are comprehensively summarized and illustrated by pathological examples. Accordingly, a novel metric method called Normal Sinkhorn Distance (NSD) is developed to evaluate the quality of vibration data. Firstly, the raw vibration data are directly transformed into time–frequency images with a tensor format and stored to ensure the following computation efficiency. Then an off-the-shelf pretrained Inception model is employed to capture their high-order data structure and amplitude-wise dependence. Finally, the NSD metric is constructed and adopted to measure the similarity between the real data and generated data in a high-dimensional feature space. Extensive results indicate that the proposed NSD provides a more reliable indication for practitioners than the existing evaluation measures on the data quality.