Seedling recruitment can be influenced by seed dispersal, conspecific density dependence, and environmental factors. These forces are variant in space. In this study, seedling recruitment was investigated by the inverse modeling method. The inverse modeling framework here was made up of two components: a conspecific effect and a declining function. Power functions (P) and constant conspecific (C) effects were tried. Two types of declining functions were tried: isotropic (I) and anisotropic (A). Thus, the combination of conspecific effect and declining function generated four candidate models: PI, PA, CI, CA. These four models were used to study the seedling recruitment of 13 species in a 20 ha forest plot in subtropical China. It was found that PI, PA, CI, CA are the best models for two, three, five, and three species, respectively. Negative exponents in P were found in three species, which may indicate negative density-dependent mortality. Among those species that supported an anisotropic component, all moderately shade-tolerant and shade-tolerant species except Calophyllum membranaceum had higher possibilities of successful recruitment if their altitudes were relatively low, consistent with their ecological niches. The shade intolerant species, Castanopsis fissa produces seeds weighing 6–250 times more than other species. Yet, its seedling recruitment was more successful at higher altitudes, which again was consistent with its ecological niche. Our research indicated that it is necessary to take anisotropic forces into account when investigating seed dispersal and seedling recruitment in regions with complex topography, and that the niche-based processes and density-dependent mortality at least play some part in constructing the seedling distribution pattern.