There have been many earlier studies of the biodiversity and ecosystem services of abandoned farmlands, but studies of abandoned villages are limited, particularly in Asian countries. The drastic aging and depopulation of Japanese society will likely lead to increased village abandonment in the future. To properly manage abandoned villages and build a comprehensive land-use plan, we need to know how natural succession will proceed after the abandonment of local towns and villages. We investigated forest structures and compositions at the sites of abandoned mining villages and compared them with ambient undisturbed natural forests. The environmental factors that determine plant succession are the distance from ambient forests, land-use period, and years since abandonment. The results showed that the sizes of tall trees and species diversity of understory vegetation recovered to the level of the control sites approximately 50–60 years after village abandonment. However, the species compositions of tall trees and understory vegetation were fairly different from those of the control sites. The distance from ambient natural forest, which regulates the seed supply for abandoned sites; land-use periods, which are surrogates for the durations of anthropogenic disturbance; and times since abandonment are strong drivers of the recovery of forest species compositions.