Composite optical measurement systems are widely used in the field of precision measurement due to their combination of inspection with high accuracy, speed, wide range, real-time, and other advantages. Whereas errors are prevalent in measurements, in order to improve detection accuracy, the systems must be compensated for geometric errors in three-dimensional space. Aiming at the complex situation of multi-probes and multi-zooms in the composite optical measurement system, the current error modelling methods are difficult to be directly applied, so this paper establishes a unified three-dimensional volumetric error model based on the theory of multi-body system and combined with the principle of geometric optics, performs the error verification through the direct measurement method, and finally realises the compensation of geometric error in the continuous space of the whole measurement range. Eventually, the accuracy of the proposed error model and the effectiveness of the error compensation method were verified by a laser interferometer and standard objects to be measured, and the integrated geometric error of the system was decreased by 76.55%, which effectively improved the accuracy of the system. The error modelling and compensation method proposed in this paper provides a new idea for the error compensation of the zoom measurement system, and at the same time, it is universal for the measurement systems of different structures and motion forms, which can be widely used in the field of precision measurement.