Large trees are essential for carbon storage and biodiversity conservation. While an increasing number of studies have focused on large trees in primary forests, little is known about them in secondary and planted forests. We surveyed 86,936 trees in secondary forests and 91,294 trees in planted forests in Zhejiang, China, to investigate the distribution patterns and determinants of large trees in these forests. We found a mean density of large trees (DBH ≥ 30 cm) of 15 ± 13 stems ha−1 in secondary forests and 11 ± 9 stems ha−1 in planted forests. Moreover, the mean density of trees with DBH ≥ 60 cm was 0.36 stems ha−1, indicating that large trees are particularly rare in secondary and planted forests. These large trees were primarily occurred in secondary forests that living in high-elevation area with less human exploitation and colder and wetter climates, and in planted forests with higher species richness and lower tree density. In addition, the density of large trees in these forests significantly increased with tree species richness and decreased with increasing tree density. These results indicate that the sparse large trees were the legacy of historical human activities in the studied area, but currently, the development of large trees is still limited by the improper forest structure characterized by low species diversity and high tree density. To better conserve large trees, there is an urgent need for enhanced conservation policies for secondary forests, such as establishing forest parks for forests with large trees, and implementing near-natural forest management practices for planted forests, which include planting mixed native tree species and maintaining moderate tree density.