This paper describes a new approach to evaluate the instability of beams on a softening elastic foundation. The basis of this approach is solving a nonlinear frequency problem. The beam is made of multiscale composite materials based on carbon nanotubes (CNTs) and long fibers. Homogenization methods are utilized in a sequential process to obtain the effective material properties of these composites. Furthermore, the thermal environment's effect on the beam's response is examined. Reddy's third-order shear deformation theory approximates the induced displacements in the composite beam. Also, the kinetic and strain energies of the beam are calculated based on the von Kármán nonlinear strains and linear thermoelasticity theories. A recently reported Lagrangian nodal weak formulation (LNWF) method is implemented to obtain the linear and nonlinear stiffness and mass matrices governing the beam's vibrations. The modified coefficients for nonlinear stiffness matrices are obtained using the residual Galerkin method across time. Finally, the nonlinear natural frequencies are calculated by employing the iterative displacement control strategy. In the results section, in addition to examining various parameters on nonlinear free vibrations, the effect of these parameters on the displacement required for the loss of stability of the beam placed on the softening foundation is also examined.