Building height is a crucial indicator when studying urban environments and human activities, necessitating accurate, large-scale, and fine-resolution calculations. However, mainstream machine learning-based methods for inferring building heights face numerous challenges, including limited sample data and slow update frequencies. Alternatively, satellite laser altimetry technology offers a reliable means of calculating building heights with high precision. Here, we initially calculated building heights along satellite orbits based on building-rooftop contour vector datasets and ICESat-2 ATL03 photon data from 2019 to 2022. By integrating multi-source passive remote sensing observation data, we used the inferred building height results as reference data to train a random forest model, regressing building heights at a 10 m scale. Compared with ground-measured heights, building height samples constructed from ICESat-2 photon data outperformed methods that indirectly infer building heights using total building floor number. Moreover, the simulated building heights strongly correlated with actual observations at a single-city scale. Finally, using several years of inferred results, we analyzed building height changes in Tianjin from 2019 to 2022. Combined with the random forest model, the proposed model enables large-scale, high-precision inference of building heights with frequent updates, which has significant implications for global dynamic observation of urban three-dimensional features.