A material platform of highly c-axis oriented Zn1-xMgxO thin films is developed for nonlinear planar waveguides and electro-optic modulators on Si. Mg content in the film greatly influences the quality of film growth. The second harmonic generation measurement and Maker-fringe analysis reveal that the second-order nonlinear susceptibility tensor element χ33 of the annealed Zn0.72Mg0.28O is approximately 4.2 times larger than that of ZnO. The propagation loss of 633 nm wavelength light in the annealed air/Zn0.72Mg0.28O/SiO2 slab waveguide is 0.68 ± 0.09 dB/cm and 0.48 ± 0.03 dB/cm for the TE0 and TM0 modes, respectively. These results suggest the great potential of the c-axis oriented Zn0.72Mg0.28O nonlinear planar waveguides for applications in on-chip optical interconnects.