For vision‐based structural displacement measurement and health monitoring, the scale factor (SF) calculation plays a pivotal role in converting the pixel displacement into the actual one. On the other hand, for the current SF calculation methods, the object distance, between the to‐be‐measured points on the object plane and the optical center of the shooting instrument, has to be measured in advance due to the existence of the pitch and horizontal angle under off‐axis measurement. Unfortunately, it is usually inefficient, difficult, and even impossible to obtain the object distances for all the to‐be‐measured points, especially for the full‐field displacement monitoring on huge scale structures. In this paper, a novel SF calculation method was proposed to calculate the full‐field scale factor of the host structure under off‐axis measurement by combining with the similarity relation of the camera imaging model. With the help of this method, all the scale factors of the to‐be‐measured points can be calculated accurately and highly efficiently, as long as the object distance of any one point or the geometric dimensions of any feature object on the object surface is provided. In addition, to quantitatively assess the SF calculation accuracy of the method, a static vision‐based measurement investigation was firstly conducted, and then, two experimental investigations about the multipoint and full‐field structural displacement measurement on a cable‐stayed bridge model and a simply supported beam model were conducted to validate the effectiveness and feasibility of the proposed method. Finally, all the results demonstrated that the proposed method exhibits an excellent performance on the SF calculation under off‐axis measurement and provides a great potential to be utilized for the vision‐based structural multipoint and full‐field displacement measurement and health monitoring.