Glasses exist because they are not able to relax in a laboratory time scale toward the most stable structure: a crystal. At the same time, glasses present low-frequency vibrational-mode (LFVM) anomalies. We explore in a systematic way how the number of such modes influences thermal relaxation in one-dimensional models of glasses. The model is a Fermi-Pasta-Ulam chain with nonlinear springs that join second neighbors at random, which mimics the adding of bond constraints in the rigidity theory of glasses. The corresponding number of LFVMs decreases linearly with the concentration of these springs, and thus their effect upon thermal relaxation can be studied in a systematic way. To do so, we performed numerical simulations using lattices that were thermalized and afterwards placed in contact with a zero-temperature bath. The results indicate that the time required for thermal relaxation has two contributions: one depends on the number of LFVMs and the other on the localization of modes due to disorder. By removing LFVMs, relaxation becomes less efficient since the cascade mechanism that transfers energy between modes is stopped. On the other hand, normal-mode localization also increases the time required for relaxation. We prove this last point by comparing periodic and nonperiodic chains that have the same number of LFVMs.