Precise measurement and calibration of the amplitude-frequency response (AFR) of coherent optical transmitters (Tx) are paramount for generating high-quality signals, especially when the transmitter has insufficient bandwidth. The previously proposed self-measurement and calibration (SMC) method based on multi-tone signals (MTS) is cost-effective, fast, and convenient. However, this method exhibits a compromised AFR measurement precision in high-frequency regions, especially when the digital-to-analog converters (DAC) of the Tx have a limited resolution. To solve this problem, we optimize the amplitudes and phases of the MTS by pre-emphasis and the sequential quadratic programming (SQP) algorithm to improve the accuracy in the high-frequency region. Numerical simulations show a notable reduction in AFR measurement error, from about 4 dB to 0.8 dB, when employing a DAC with a four-bit resolution. Additionally, experiments utilizing a 22 GHz 3-dB bandwidth transmitter to generate 61 GBaud dual-polarization 16QAM signals demonstrate a 60% BER reduction. The proposed SMC method is attractive for high-baud rate systems employing Tx with limited bandwidth and DAC resolution.