Fault diagnosis is efficient to improve the safety, reliability, and cost-effectiveness of industrial machinery. Deep learning has been extensively investigated in fault diagnosis, exhibiting state-of-the-art performance. However, since deep learning is inherently uninterpretable, the low trustworthiness of the diagnostic results given by these black-boxes has always been a limiting factor in industrial applications. Specially, the monitoring data under unforeseen domains will be easily misdiagnosed without any symptoms. To address this issue, this paper explores the fault diagnosis in a probabilistic Bayesian deep learning framework by exploiting an uncertainty-aware model to understand the unknown fault information and identify the inputs from unseen domains, ultimately achieving trustworthy diagnosis. Moreover, the diagnostic uncertainty is decomposed in two aspects: (1) epistemic uncertainty, reflecting the discrepancy of test input relative to the training data, and (2) aleatoric uncertainty, referring to the noise originating from the input, offering a deep understanding of the unknowns in the diagnostic model. The proposed framework not only can accurately identify the faults belonging to a known distribution, but also provides insights into uncertainty and avoid the erroneous decision-making. Last, but not least, comprehensive diagnostic experiments considering unseen scenarios are used to demonstrate the effectiveness of proposed framework, providing competitive results.