Compressed Sensing (CS) is an effective method for improving the real-time performance of crack-induced acoustic emission (AE) signals analysis in Structural Health Monitoring (SHM) of rails. Aiming to promote the reconstruction accuracy and speed of CS, a reconstruction method is proposed based on improved Sparsity Adaptive Matching Pursuit (SAMP) and modular dictionary update. In the proposed method, a multiscale-modular dictionary is devised based on a multiscale dataset to enhance the real-time performance of reconstruction. Meanwhile, the step size of the SAMP is adaptively adjusted by the kurtosis residuals, which promotes the reconstruction accuracy. Furthermore, to optimize the adaptability of the dictionary, kurtosis-deviation is utilized to update the dictionary adaptively and modularly. The proposed method was verified by tensile tests. The results demonstrate that the proposed method has a higher reconstruction accuracy and speed than other methods, which can guide real-time crack-induced signal analysis in the SHM of rails.