Pasture-based goat operation is popular among small-scale farmers; however, it may be challenging when forage availability is limited. Grazing opportunity can be increased by developing silvopastures after thinning pine plantations. To successfully manage silvopastures, it is important to understand animals’ interaction with trees. The objective of this study was to evaluate the use of goats in the southern pine silvopasture system. Studies were conducted at two sites: Atkins Agroforestry Research and Demonstration Site, Tuskegee (Site 1), and Plantersville Silvopasture Demonstration Site, Plantersville, Alabama (Site 2). Site 1 consisted of six acres of longleaf (Pinus palustris Mill.) and loblolly (Pinus taeda L.) pine trees (10–11 years old, 147 ± 21.1 trees/acre; longleaf 52.14%, loblolly 47.86%). Site 2 comprised of 14 acres of 17–18 years old loblolly pine silvopastures. Both sites were planted with cool- and warm-season forages, and plots were rotationally grazed with goats (Site 1—Kiko wethers; Site 2—mixed breed of goats) during the 2015–2016 grazing seasons. Goat performance (live weight, body condition score, and FAMACHA score) was monitored every 2 weeks in Site 1, and trees were inspected for possible damage in both sites. Goats performed better while grazing cool-season forages during the spring. In Site 1, goats debarked pine trees, especially longleaf pines during the cool-season grazing period. No debarking was observed in Site 2. This study showed that southern pine silvopastures could provide a good grazing opportunity, but trees can be vulnerable to goats even when they are 10–11 years of age.