Preventing resonance in boring bar vibration is the key to achieve high quality and efficiency. In this paper, using the three-dimensional nonlinear dynamic model of boring process, the cutter bar is simplified as a rotating non planar bending axis, starting from the constitutive relationship and stress displacement relationship of composite materials. Based on Hamilton principle, the nonlinear motion differential equation of cutting process is established. The nonlinear partial differential equations of bending vibration are discretized by Galerkin method. Using the multi-scale method, the frequency response functions of the system in the primary resonance and super resonance are obtained. Using this model, the effects of the length diameter ratio of nano-carbon materials and the content of nano-carbon materials on the vibration frequency response of the boring bar are studied. The results show that the steady vibration amplitude of the cutter bar can be reduced by adding nano-carbon materials to the fiber reinforced polymer composite (FRP) beam.