Trees in the urban environment are affected by a range of place-specific environmental and cultural factors. Nevertheless, many factors interact with each other and produce high variabilities in urban forest structures, spatial configurations, and growing conditions within the urban fabric. The variabilities and relationships among these factors require a modelling framework that incorporates variables related to the spatio-temporal information of individual trees and site characteristics. Using roadside trees in a residential district in Hong Kong, this study examined their species compositions and spatial distributions among zones with various development histories. Structural equation modelling was applied to investigate the interactions among variables concerning the physical environment, temporal changes, species selection, spatial patterns, and tree sizes. A few abundant species dominated the urban forest, with uneven distributions of roadside tree densities and species-specific clusters found within the study area. The empirical results showed (i) the physical environment influenced the existence of roadside trees, the occurrence of specific species and tree growth; (ii) species selection mediated the influences of the physical environment on tree sizes and spatial patterns; (iii) despite the linkage between existing tree patterns and development histories, replacement plantings further modified the compositions and reflected a shift in preferences; (iv) the proximity to nearby urban structures and the spatial properties among planting locations were important factors in spatial planning. Hence, urban forest managers play a crucial role in mediating the interactions among physical drivers, planting histories, and tree performances. It is recommended that species diversity, especially native trees, should be promoted within the study area, while planting strategies matching site dimensions, species and attainable sizes should also be prioritised. By generalising the relative effects that determined the existing patterns of urban forests, the modelling framework can provide a strategic reference for target-oriented tree management in Hong Kong and other cities.