Clarifying the critical driving factors and the driving mechanism for building carbon emissions is fundamental for carbon peak and carbon neutrality prediction. This study comprehensively identifies 23 representative factors using the literature research and questionnaire survey method. Then, by coupling the decision-making trial and evaluation laboratory (DEMATEL) method and the interpretive structural model (ISM), an integrated DEMATEL-ISM model is developed. The proposed model is further employed to quantify the causal relationships among the factors and establish a multi-level recursive structure model to determine the action mechanism of those drivers. The results show that the 11 cause factors, 12 effect factors and the three most important factors including the economic development level, building end-use energy consumption and end-use energy intensity are identified based on the calculated cause and center degree. Based on the obtained hierarchy, the 23 factors are divided into six levels, among which the population factors, the economic development, the urbanization level and the share of the tertiary sector are the root factors leading to the building carbon emissions. In comparison, policy factors are the most critical inhibiting factors. This study can not only promote the theory and model development for the driving mechanism of carbon emissions, but also help decision-makers develop scientific carbon reduction strategies.