The geometrically nonlinear vibrations of beams with rectangular cross section, which may experience longitudinal, torsional and bending deformations, in any plane, are investigated by the p-version finite element method. Two main models are used: one of them is based on Bernoulli–Euler theory for beam bending and the other is based on Timoshenko theory; both models assume that the cross section rotates as a rigid body and is free to warp in longitudinal direction, as in Saint-Venants’s theory for torsion. The geometrical nonlinearity is taken into account by considering Green’s strain tensor and the importance of the longitudinal displacements of quadratic order, which are most often neglected in the strain–displacement relation, is here examined. Generalized Hooke’s law is used and the equation of motion is derived by the principle of virtual work. The importance of the warping function is analysed for different rectangular cross sections, and it is shown that its consideration can be fundamental to obtain correct results. Then, it is shown that the linearization of the trigonometric functions related with the twist angle, which is usually applied in the displacement field in models based on Saint-Venant’s hypothesis, should be done in the strain–displacement relations. Comparisons of the models for 3D beams based on Bernoulli–Euler’s and Timoshenko’s theories is presented. It is shown that if the rotations along the transverse axes are directly approximated by the respective derivatives of the transverse displacement functions, as is assumed in the model presented here that is based on Bernoulli–Euler’s theory, the additional shear stresses that appear when the bending and torsion motions are coupled, lead to wrong results. Finally, taking into account accuracy and simplicity, a model is chosen and employed to investigate the nonlinear forced vibrations of beams using direct integration of the equations of motion in the time domain. Examples of bending–torsion couplings due to the nonlinear terms are presented in dynamical problems.