Understanding the characteristics of downhole vibration helps optimize the drilling parameters and improve the rate of penetration (ROP). This work establishes a nonlinear drillstring–bit–rock model to predict the downhole vibration behavior of 3D curved wells. The main novelty of this model is to consider the complex nonlinear bit–rock interaction and well trajectory. The model is solved by the Newmark method and verified by an indoor experiment and field data. The drillstring dynamics and effects of the main working parameters on the vibration characteristics are obtained and analyzed. Results show that, in the horizontal section, the drillstring motion and shape deformation are stable, and radial vibration is dominant. In the buildup section, the drillstring motion is irregular and messy, shape deformation becomes large, and radial vibration is significant but tangential vibration starts to become intense. In the vertical section, the drillstring motion is small in wellbore center, shape deformation is stable, and vibration severity reduces. The weight on bit (WOB) only affects drillstring vibration near the bit, and radial, tangential, and axial accelerations intensify as WOB increases. RPM has little effect on the drillstring vibration in 3D curved wells. In the buildup and horizontal sections, as the coefficient of friction (COF) increases, the radial and tangential accelerations rise while the axial vibration obviously weakens. When drilling in 3D curved wells, small WOB and large RPM can be adopted to reduce downhole vibration and increase ROP. The safety of drillstring and downhole tool also needs more attention when drilling in formations with large COF, such as gravel stone and sandstone. Large oil/water ratio or lubricant is recommended to improve the lubricity of drilling fluid. The research results could provide a valuable and useful guidance for drilling engineers to improve ROP and reduce downhole accidents.