In this paper, we propose a three-dimensional finite element model that can simulate the caisson foundation behavior in gravelly soil. The soil model adopts a combination of the porous elastic model and the modified Drucker–Prager/Cap model for simulating the gravelly soil. For avoiding problems in laboratory testing on gravel specimens, a practical approach is proposed to determine and calibrate the parameters of the soil model using results of field geotechnical tests, such as a shear wave velocity profile, strength parameters of direct shear tests, and pressure–displacement curves of vertical and horizontal plate loading tests. The proposed finite element model was applied to simulate in situ lateral load tests on bridge caisson foundations in gravelly soil. Using the calibrated soil model, responses of caisson displacement, caisson head rotation, and soil displacement analyzed were observed to be in agreement with the measured responses. Caisson and surrounding soil responses were also investigated in detail.