Nonlinear geometries and cross-sectional asymmetry can be among the most critical contributing factors affecting the performance and stability of composite shafts. The influence of these factors, which should be devoted to particular attention in the design of these systems and have not been investigated yet, are evaluated analytically in this study. The shaft is simply supported, made of orthotropic multi-layers, and spinning at a constant speed. To express the nonlinear system’s behavior, which is due to the large amplitude of the vibrations, it is assumed that the shaft is under the stretching assumption. Moreover, a rectangular cross-section is used to model the asymmetry that results in parametric excitation in the system. To accurately investigate the behavior of composite material, an optimal lay-up is employed. The gyroscopic coupling is included because of Rayleigh beam theory, and Euler’s angles are employed to achieve the angular velocities. The analytical study of the parametrically excited system, obtained by the method of multiple scales, is performed in two categories of resonant and nonresonant cases. In the nonresonant case, the analytical investigation suggests that the asymmetric shaft behaves like a symmetric one therefore, the parametric excitations do not have a significant impact. This claim is confirmed by numerical results. Also, the presence of the gyroscopic coupling and hollowness of the shaft causes the beating phenomenon in the system. However, in the resonant case, the presence of parametric excitation plays a pivotal role. The results also show that under certain conditions and despite the presence of damping, asymmetric balanced rotors can have a nontrivial stable amplitude. This response exists as long as the parametric excitation effects dominate the damping effects. Although damping reduces the vibrations’ amplitude, it can improve the stability of the system and eliminates unstable responses. Furthermore, the time response and frequency response curve of the system is carefully evaluated for various geometric design parameters and operation speed. Depending on the operating speed, the system can experience supercritical or subcritical pitchfork bifurcation. In addition, a detailed description of the system’s Campbell’s diagram, damping effect, and bifurcation is provided. Furthermore, it is proved that internal resonance cannot occur in the system. The accuracy of the analytical responses of the resonant case is compared with numerical ones. Stability of trivial and nontrivial responses is discussed in the time response and phase portrait of the system simultaneously. Finally, in the undamped system, multi-frequency responses are appeared as homoclinic and heteroclinic closed orbits.