Increasing reports indicated that multiscale entropy (MSE) is an efficient approach to investigate various physical and physiological states, especially effective for cardiac electrophysiology and locomotive activity. However, the feasibility of applying MSE to the fast oscillation systems, such as electroencephalogram or magnetoencephalography, may be impeded by the rapid patterns of the brain signals. To this end, the amplitude-modulated MSE is proposed. Simulations include Gaussian white noise, 1/f noise, stationary and fractionally integrated autoregressive processes, and logistic map. The proposed framework is demonstrated that capable of identifying chaotic patterns from random distributions. For time series with periodic patterns, AM-MSE levels maintain zero, while the AM-MSE curves of the chaotic dynamics increase first and then decrease gradually across scales. In the AM-MSE curves, referenced to the MSE curves of the real brain signals, the time scales of the spectral peaks are relocating to the relatively lower bands for all sleep stages, resulting in a prolonged ramp-up band. Significant differences were found in both the corresponding areas under the curves (AUCs) and slopes of the raw signals per se and their amplitude modulations among sleep stages in multiple frequency bands (p<0.0001). Compared to the raw signals, the MSE curves of their amplitude modulations show relatively stable tracks and clearer hierarchical layers in particular. We suggest that the proposed amplitude-modulated MSE is an effective tool to investigate the complexities of fast oscillatory activities like brain signals.