This paper presents the generation and the performance evaluation of optical minimum shift keying (MSK) for high-speed and high-spectral-efficiency wavelength-division multiplexing systems. The detailed principle on optical MSK generation and detection is derived and analyzed. Optical MSK data (10.7 Gb/s) are successfully generated by an experiment using the proposed scheme. The results show that the optical MSK signal exhibits a very compact optical spectrum. The performance of the optical MSK modulated system is evaluated and compared with those of 50% duty-cycle return-to-zero differential phase shift keying (RZ-DPSK) and 50% duty-cycle return-to-zero on-off keying (RZ-OOK) modulated systems, via both simulation and experiment, in terms of dispersion tolerance, linear crosstalk, self-phase modulation (SPM) tolerance, stimulated Brillouin scattering (SBS) threshold, optimal receiver filter bandwidth, and optimal link launch power into single-mode fibers and dispersion compensation fibers. Experimental and simulation results show that the MSK signal exhibits the largest tolerance against fiber dispersion and SPM effect as compared with RZ-DPSK and RZ-OOK. Experimental results also confirm the negative penalty against the SPM effect. The SBS threshold of both the MSK and RZ-DPSK systems is more than 10 dB higher than that of the RZ-OOK system. The MSK that is generated with conventional carrier-suppressed return-to-zero (CSRZ) pulses exhibits a narrower optimal optical filter bandwidth but a wider electrical filter bandwidth compared with those of RZ-OOK and RZ-DPSK. The MSK signal that is generated with pure dual-mode pulses has a similar optimum range to RZ-OOK and RZ-DPSK at a high optical signal-to-noise ratio (OSNR) but has a narrower optimal optical filter bandwidth at a low OSNR. The optimum fiber launch power study shows that the MSK signal that is generated using the conventional CSRZ pulses exhibits higher optimum launch power, which agrees with the SPM tolerance study.