Quantum information processing using atomic qubits requires narrow linewidth lasers with long-term stability for high fidelity coherent manipulation of Rydberg states. In this paper, we report on the construction and characterization of three continuous-wave (CW) narrow linewidth lasers stabilized simultaneously to an ultra-high finesse Fabry-Perot cavity made of ultra-low expansion (ULE) glass, with a tunable offset-lock frequency. One laser operates at 852~nm while the two locked lasers at 1018~nm are frequency doubled to 509~nm for excitation of $^{133}$Cs atoms to Rydberg states. The optical beatnote at 509~nm is measured to be 260(5)~Hz. We present measurements of the offset between the atomic and cavity resonant frequencies using electromagnetically induced transparency (EIT) for high-resolution spectroscopy on a cold atom cloud. The long-term stability is determined from repeated spectra over a period of 20 days yielding a linear frequency drift of $\sim1$~Hz/s.