Swidden agriculture as a production system has been practiced for thousands of years, based on ecological processes of forest ecosystems. While this cultivation system has often been framed as contributing to environmental degradation, it can also be argued to be compatible with sustainable agricultural practices. Its sustainability strongly depends on the length of the recovery period, the period of secondary forest succession, after swidden agriculture. In this study we focused on soil fertility maintenance and recovery during secondary succession after swidden agriculture, compared to old-growth forests. Smallholders practiced a moderate form of swidden agriculture, abandoned the production field after one or two production cycles, and returned to the field after 10–15 years. Sampling took place in two traditional communities situated in the southeast portion of the Atlantic Rainforest at São Paulo state, Brazil. Soil samples were collected from forest fragments belonging to seven different Caiçara families (descendants from Amerindians, African Brazilian and European colonizers). Our chronosequence spanned a period of sixty years, while the old-growth forests were 100 years old. In all 28 sites were sampled, of which 19 were from ombrophilous forests on clay and 9 from restinga vegetation on sandy soil. Soils were analyzed for texture, macro- and micronutrient availability, bulk density, pH and nitrogen and carbon stocks. Linear regression, with clay content and fallow period as main factors, showed that successional age of the forest stand had only a significant effect on pH and Mn. Clay content of the soil influenced availability of nutrient bases and the CEC. Other soil properties, including stocks of carbon and nitrogen were not influenced by fallow period. Our data indicate that swidden agriculture might not lose key soil nutrients over time and hence can be considered an ecologically sustainable system in the region, challenging the negative perception of swidden agriculture.