Abstract The common utilization research methods in mining engineering include theoretical analysis, mechanical modeling, similarity model test, numerical simulation, and field test. And physical similarity model test is one of the main research methods for mining engineering problems. Underground engineering is often in the complex three-dimensional stress state. Compared with plane model, three-dimensional model can actually reflect the stress state of surrounding rock. Nevertheless, traditional measurement methods can’t achieve internal deformation of the model with multi-scale distributed monitoring. Optical fiber sensing technology provides a solution to these problems. The sensing fibers are arranged in the plane model with the size of 3000 × 200 × 1280 mm and the three-dimensional model with the size of 3600 × 2000 × 2000 mm, and the strain distribution of the model has been analyzed with the consideration of different positions relationship between the working face and the optic fiber. The results show: the strain coefficient of the test optic fiber with 2 mm diameter calibrated by the uniform strength beam experiment is 0.0497 MHz/μe. The frequency shift of the optic fiber is positive when the fiber is under tension state, and the result is opposite when under compression; In physical model tests, when the working face is close to the sensing fiber, the strain curve shows a negative step change caused by the abutment pressure. When the working face cross the fiber, the strain curve shows a positive step change due to the downward movement of broken rock layer; after the working face away from the fiber, the re-compaction of the broken rock block induced by gravity load causes the optical fiber to be under pressure, and the sensing fiber generates compressive strain. The three-segment distribution area of the strain curve corresponds to the “three-zone” height range of the overburden caused by mining, respectively. The test results could provide theoretical guidance for the application of distributed optical fiber in the determination of the caving zone and fractured zone range of overburden induced by coal mining.