Prefabricated buildings have the characteristics of being energy saving, protecting the environment, being labor saving and having a fast construction speed, which meet the requirement for the sustainable development of the construction industry and are been strongly supported by the country. However, the promotion of prefabricated buildings in China is still in its infancy, with numerous barriers, and the correlation between these factors is not well tested. Therefore, the objective of this study was to identify the main obstacles to the promotion of prefabricated buildings, clarify the correlation between the main barriers and classify them, so as to propose targeted promotion strategies. Combined with the PEST theory and the technology–organization–environment framework, this study has established a list of 27 barriers including policy, the economy, society, technology and management, through a literature review. Experts were invited to complete a questionnaire survey to assess the relative importance of barriers, and 15 major barriers were identified based on the analytic hierarchy process method. We conducted semi-structured interviews, determined the direct relationship between various barriers, established the adjacency matrix and calculated the reachability matrix, established the multi-level hierarchical structure model based on interpretative structural modeling, used Matrices Impacts Croises–Multiplication Appliance Classement to classify factors, and then proposed the industry–organization–project three-level promotion strategy according to the classification results. The results showed that technical factors accounted for the largest proportion in the list of barriers, followed by society, the economy, political factors and, finally, management factors. The main obstacles to the popularization of prefabricated buildings can be divided into three categories: autonomous variables, dependent variables and driver variables. It is necessary to put forward countermeasures from different dimensions to promote the application of prefabricated buildings. This study provides a practical reference for future research, attempts to provide solutions to promote the popularization of prefabricated buildings and also provides methodological guidance for similar research.