Fault diagnosis of wheelset bearings in high-speed trains has attracted constant interest in the scientific community and industrial field. Under the harsh working condition, e.g., time-varying speed and load, most existing methods are hindered by the limited and unknown situations of wheelset bearings. Although the self-calibrated convolution is proven to effectively expand the receptive field with more accurate discriminative regions, its use in fault diagnosis still lacks needed physical interpretation as well as computational efficiency. To this end, this article presents a novel framework by using the logarithmic short-time Fourier transform and the modified self-calibrated convolution. It first manifests a time-frequency map that has explicit physics meaning while reducing the gap between high energy and detailed characteristics in the masking of interfering signals. To simplify redundant kernels, a modified self-calibrated residual block is proposed without introducing any more parameters, while preserving an interpretable and simple structure. The effectiveness and robustness of the proposed method are verified by the experimental data collected from an industrial railway axle bearing test rig. Results are found superior to those of five state-of-art methods, which are more practical in terms of accuracy, cost time, and model size.