Spectral amplitude modulation (SAM) method, as an automated and empirical nonlinear filtering approach, has shown great promise for rotating machinery fault diagnosis. However, due to the inherent shortcomings of Fourier transform in SAM, leading to significant errors in the edited amplitude, also it is easy to fail with selecting the optimal weight value manually under intense background noise. To solve the aforementioned drawbacks, the Stockwell transform spectral amplitude modulation (STSAM) method is proposed. The Stockwell transform (S-transform) is first utilized to obtain the phase and amplitude with time–frequency information. Then, the edited signals can be reconstructed by inverse S-transform with the above actual phase and the modified amplitudes under different weights. Hence, more comprehensive and accurate information about the amplitude can be computed. After that, their normalized square envelope spectra under each cyclic frequency are calculated to showcase the fault characteristics. Moreover, a novel indicator is proposed to automatically choose the optimal weight in STSAM, thus clearer characteristic frequencies can be represented by the optimal square envelope spectrum (OSES). Finally, the effectiveness and superiority of the STSAM and OSES methods are systematically demonstrated by the comparative studies with the SAM and time–frequency SAM approaches using simulated signals and real-world datasets.