It is shown that a simple system without a cavity exhibits optical bistability with no hysteresis but with symmetry-breaking or pitchfork bifurcation. The optical system is composed of a cell containing atoms with spin in the ground state, a $\frac{\ensuremath{\lambda}}{8}$ plate and a mirror, and an incident light beam linearly polarized and nearly resonant with the atomic absorption line. The system has a positive-feedback loop for the rotation of polarization through competitive optical pumping by ${\ensuremath{\sigma}}_{\ifmmode\pm\else\textpm\fi{}}$ circularly polarized components. When the intensity of the incident light exceeds a critical value, symmetry breaking occurs and its polarization is subject to rotation in the clockwise or anticlockwise sense. Correspondingly, atomic spin polarization is produced spontaneously in a direction parallel or antiparallel to the incident beam. This new type of optical bistability is found to be explained well in context with the cusp catastrophe, as well as the ordinary one with hysteresis. It is also found that, in the presence of a transverse magnetic field, self-sustained spin precession occurs, which results in the steady-state modulation of the light polarization at about the Larmor frequency. Experimental evidence of the symmetry breaking and pitchfork bifurcation is obtained by using sodium vapor.