Ex situ seed conservation is a widespread and powerful tool for the preservation of plant genetic resources. However, long-term storage of short-lived seeds is challenging in standard seed bank temperatures (4 and −20 °C), and cryopreservation has been acknowledged as the only effective option for their long-term conservation. However, as seed aging has been demonstrated during cryostorage, the potential and limits of LN storage of short-lived seeds need to be evaluated. We have examined the viability and growth of seeds from species with short-lived seeds stored up to 20 years in CREW’s CryoBioBank®, including Plantago cordata, Populus deltoides, several Salix species, and two species in the genus Betula. We have found survival in short-lived seeds after >14 years of LN storage (−196 °C). P. cordata seeds did not show a significant decrease in germination over this time. However, Salix sp. and P. deltoides accessions tested showed a final germination ranging between 0% and 96% of initial germination. Seeds of P. deltoides stored at 4 and −20 °C did not show any survival after the same storage period. Seed viability in P. deltoides and Salix sp. was overestimated when measured as total germination, rather than normal germination. Furthermore, 38–43% of seedlings from normal seedlings of P. deltoides did not continue normal development during early growth of the young plants. In addition, initial seed quality was correlated with seed longevity. These results suggest that: (1) monitoring of short-lived seeds cryostored as in this study (at 8–10% moisture) should be done at 2–5 year intervals, similar to the recommendation by IPGRI for seeds stored at −18°C, and regeneration should not exceed 10–15 years; (2) in order to minimize loss of seed quality prior to cryostorage, processing of short-lived seeds after harvest must be fast and gentle; (3) based on losses during early seedling growth and development, it is very important not to exceed the time to reach 85% of initial normal germination when planning regeneration of Populus and Salix collections to avoid large genetic erosion.