Abstract The Rydberg atom-based receiver, as a novel type of antenna, demonstrates broad application prospects in the field of microwave communications. However, since Rydberg atomic receivers are nonlinear systems, mismatches between the parameters of the received amplitude modulation (AM) signals and the system's linear workspace and demodulation operating points can cause severe distortion in the demodulated signals. To address this, the article proposes a method for determining the operational parameters based on the mean square error (MSE) and total harmonic distortion (THD) assessments, and presents strategies for optimizing the system’s operational parameters focusing on linear response characteristics (LRC) and linear dynamic range (LDR). Specifically, we employ a method that minimizes the MSE to define the system’s linear workspace, thereby ensuring the system has a good LRC while maximizing the LDR. To ensure the signal always operates within the linear workspace, an appropriate carrier amplitude is set as the demodulation operating point. By calculating the THD at different operating points, the LRC performance within different regions of the linear workspace is evaluated, and corresponding optimization strategies based on the range of signal strengths are proposed. Moreover, to more accurately restore the baseband signal, we establish a mapping relationship between the carrier Rabi frequency and the transmitted power of the probe light, and optimize the slope of the linear demodulation function to reduce the MSE to less than 0.8×10-4. Finally, based on these methods for determining operational parameters, we explore the effects of different laser Rabi frequencies on system performance, and provide optimization recommendations. This research provides robust support for the design of high-performance Rydberg atom-based AM receivers.