Although deep learning techniques have been widely used in machinery fault diagnosis, the key issue of feature learning under multiple non-ideal conditions, i.e., strong noise interference, inconsistent data distributions and class imbalance, is not tackled effectively. Thus, a novel deep neural network (DNN) named temporal self-supervised domain adaptation network (TSSDAN) is developed for machinery fault diagnosis in this article. Firstly, a temporal self-supervised learning (TSSL) method is implemented to tackle the issue of strong noise interference and class imbalance. Secondly, a specific feature extractor, perception auto-encoder (PAE) is implemented for signal feature learning by mimicking the working mode of the biological visual system and spinal reflex system. Thirdly, a cascaded domain adaptation (CDA) method is constructed to tackle the domain shift issue by considering the feature distribution discrepancy of various domains for all perception levels. Two cases (i.e., rotor and gearbox) are used in this study to evaluate the performance of TSSDAN. The experiment results illustrate the outperformance of TSSDAN in solving the problem of deep feature learning and machinery fault diagnosis under multiple non-ideal conditions.