To ensure the safe construction of prefabricated buildings and improve the efficiency of the safe evacuation of construction personnel after a fire caused by improper operation during construction, this study used the PyroSim software to numerically simulate a fire situation based on the size and volume of a prefabricated building construction site. The variation rules of smoke visibility, CO concentration, and ambient temperature in the construction site of prefabricated buildings were analyzed and the available safe evacuation time was determined. Moreover, the Pathfinder software was used for simulation in combination with the physical attributes of personnel, evacuation speed, and personnel proportions. The time required for safe evacuation was determined and the factors influencing the evacuation time, such as the quantity and location of stacked prefabricated components, machinery, and appliances, and the number of on-site construction personnel, were analyzed. The data collected by the temperature sensor, CO concentration sensor, and visibility sensor reveal that the visibility and crash time are the key factors restricting the efficiency of personnel avoidance and evacuation. At 400 s, the visibility at the escape exit of the prefabricated apartment construction site was lower than 5 m. The crashing time of the building was 360 s, which is the critical point for casualties. The first emergency evacuation simulation took 398.7 s. The required safe evacuation time (TREST) > available safe evacuation time (TASET), and the original site layout cannot facilitate the safe evacuation of all construction workers. The evacuation time can be effectively reduced by re-planning the stacking positions of prefabricated construction site components, construction equipment, and other items, and reducing the number of personnel in the construction plane. The results of the second simulation reveal that the safe evacuation time (TREST) is 355.2 s. Because it is required that the safety evacuation time (TREST) < available safe evacuation time (TASET), the results are in line with the emergency evacuation requirements. The findings of this study can provide a theoretical basis for the rational planning of evacuation passages at the construction sites of prefabricated buildings and assist the management of construction site safety.