On-site assembly of prefabricated components is crucial for the construction efficiency and quality of prefabricated buildings. In this process, a prefabricated component is first lifted and transported to the vicinity, where it will be installed using a crane. Subsequently, multiple workers collaborate to align the components with the target installation area. However, this manual installation method is time-consuming and unreliable in accuracy. A vision-based robot-assisted component installation system is proposed in this study. After the initial alignment using a crane, the two robots cooperate to adjust the position and orientation of the component accurately. This automated assembly method consists of the following tasks: (1) The target installation pose of the component, which can be used as a guide for the movements of robots, is extracted from the building information model (BIM) based on the industry foundation class standard. (2) An optical-marker-based method is applied to position the robots in the coordinate system of the site and identify the pose of the reserved hole in the lifting prefabricated component. Based on this pose, the robot is manipulated to thread the end effector into the reserved hole. (3) A planned working path of the end effector in Cartesian space combined with the current pose of the end effector and the target installation pose is obtained. (4) The end effector moves along a predetermined path to drive the component precisely. Finally, the accuracy, feasibility, and effectiveness of the robotic automated construction method are verified through installation tests on a precast panel.