Bistable composite shells with high storage efficiency have great potential applications in deployable space structures. This paper proposes a constrained vibration optimization approach for improving the bending-mode frequency of cantilevered bistable reeled composite shells (BRCS) with respect to fiber orientation angles using a machine learning (ML) method. First, the bistability and a specific coiled diameter are considered as constraints to classify the input data. The data set of the support vector regression (SVR) model is constructed based on the finite element (FE) simulation results, followed by constricted particle swarm optimization (PSO) to improve the bending-mode vibration frequency of a cantilevered BRCS. A 15.5% improvement of the vibration frequency with respect to the benchmark is achieved at α=59.3° and β=32.2°, which maintains great consistency with published results. Additionally, the optimization approach based on ML is further utilized to improve the vibration frequency of BRCS subjected to constraints of constant arc length and coiled diameter. The vibration frequency is improved by 85.3% with respect to the benchmark shell with optimized parameters of R= 16 mm, γ=358°, and the stacking laminate sequence of [60/80/0/−80/−60]. Evaluation and validation analyses of the ML model demonstrate that vibration optimization using ML yields high computing efficiency and accuracy. This optimization approach has great potential in real-life engineering applications.