Nanobeams and nanowires are widely used as building blocks in the rapid development of Nano/Micro-electro-mechanical system (N/MEMS), micro-sensors, energy harvesting and storage devices, etc., and their vibration behaviors have aroused great concerns in both pure science and engineering applications. In this study, we investigate the nonlinear free vibration of a nanobeam considering its surface effects, including the surface elasticity and residual surface stress. Firstly, a mechanics model on the transverse vibration of a cantilever nanobeam is developed according to the Hamilton's principle. In use of the Galerkin and complex normal form methods, the approximate analytical solution of the nonlinear equation is obtained, which has been corroborated by the numerical simulation. Next, the effects of residual surface stress on the nonlinear dynamic behaviors of the system are examined both theoretically and numerically, which indicate that surface effects have great impact on the quasi-periodic and chaotic motions of the system. The present work can provide a theoretical basis for the precise design of nanowires or nanofibers in atomic force microscopy, generators and nano-sensors in electronic devices.