Reflection second-harmonic generation from the polished waveguide end face is used to investigate the second-order nonlinear optical properties of as-exchanged and annealed proton-exchanged (PE) waveguides in different H/sub x/Li/sub 1-x/NbO/sub 3/ phases. A detailed correlation is done between the nonlinear properties, the processing conditions, the refractive index changes, and the optical losses of the waveguides. It is found that for the direct PE samples, where the /spl beta//sub 4/, /spl beta//sub 3/, and /spl beta//sub 1/ phases are generated at the surface, the nonlinearity in the guide is strongly reduced by more than 85% of its bulk value, while for waveguides prepared in the /spl beta//sub 2/ phase, the nonlinear coefficient is about 55% of the bulk one. A consequence is that the step-like /spl beta//sub i/-phase PE LiNbO/sub 3/ waveguides with large refractive index increase are advantageous for efficient SHG in Cherenkov configuration. The nonlinearity, strongly reduced after the initial proton exchange, is found to be restored and even increased after annealing. However, this apparent increase of the nonlinearity is accompanied by a strong degradation of the quality of the second-harmonic generation reflected beam in the region of initial waveguides due to beam scattering. The graded proton exchange technique and dilute melt proton exchange have been shown to produce high-quality waveguides with essentially undergraded nonlinear optical properties. It has been also shown that the nonlinear properties of annealed proton exchanged LiNbO/sub 3/ waveguides can be effectively recovered by the reverse proton exchange technique. The results obtained are important for the design, fabrication, and optimizing of guided-wave nonlinear optical devices in LiNbO/sub 3/.