As early as the mid-18th century, Euler proposed a critical load formula for compression rods with multiple half-sine wave modes. However, a critical load formula for compression rods constrained by cylinders with multi-helix buckling modes has not yet been proposed. In this paper, a mechanical model of multi-helix buckling of a slender compression constrained by a cylinder is established. Utilizing the dynamic relaxation method, a finite element solution strategy is presented. Its notable advantage lies in its capability to compute stable configurations of multiple helical buckling, effectively addressing challenges such as contact identification and convergence issues during computation. Furthermore, parameter analysis quantifies the influence of length, diameter, and annular gap on the critical load, while also analyzes the shear force, bending moment, and contact force after buckling. This paper presents formulas for the critical load of multi-helix buckling and the length of the multi-helix buckling modes, which is suitable for various length, diameter, and annular space gap conditions, and partially validates these results against finite element analysis.