The size structure of stony coral populations can reveal underlying demographic barriers to population growth or recovery. Recent declines in coral cover from acute disturbances are well documented, but few studies have assessed size structure and the demographic processes that determine population growth. Vital rates, such as recruitment and survival, vary spatially and temporally in response to environmental conditions, in turn influencing assemblage composition. The Southeast Florida Coral Reef Ecosystem Conservation Area (Coral ECA) is a high-latitude reef system offshore of a heavily urbanized coastline. Consecutive heat stress events, stony coral tissue loss disease (SCTLD), and Hurricane Irma caused significant declines in stony coral cover and density from 2014 to 2018. The recovery potential of stony coral assemblages is influenced by their composition, the size structure of the remnant populations, and population growth during inter-disturbance periods. To assess the viability of the remaining stony coral assemblages in the Coral ECA, we quantified variation in stony coral recruit density, abundance, size structure, and assemblage composition across depth and latitude at permanent sites over 3 years (2019–2022) when no disturbances occurred. We found spatial decoupling in recruit density, adult colony density, and cover that maintains a preponderance of small colonies and skewed size structure. At sites close to shore where recruit density was higher, there was limited evidence of survival and growth of recruits, while at sites where large colonies were sampled or cover was relatively high, there was limited recruitment. The majority (>75%) of recruits sampled were Siderastrea siderea, but size frequency distributions were positively skewed and the coefficient of variation was high, suggesting high recruit/juvenile colony mortality and little growth into larger size classes. Porites astreoides size structure was generally lognormal and mesokurtic, particularly closer to shore, suggesting a transition between size classes. Skewness decreased moving offshore in Montastraea cavernosa and S. siderea, suggesting a transition between size classes. Recruit and adult diversity also increased moving offshore, but recruits of most species were uncommon throughout the study area. We suggest that low recruitment and high mortality, particularly in small colonies and inshore, even during inter-disturbance periods, limit the population growth of stony coral assemblages in southeast Florida.