In the present paper, a fluid-particle coupling method is directly derived from the Navier-Stokes equations (NSE) by applying the concept of volume-filtering, yielding a physically consistent methodology to incorporate solid wall boundary conditions in the volume-filtered flow solution, thereby allowing to solve the governing flow equations on non-body conforming meshes. The resulting methodology possesses similarities with a continuous forcing immersed boundary method (IBM) and is, therefore, termed physically consistent IBM (PC-IBM). Based on the recent findings of Hausmann et al. [1], the closures arising in the volume-filtered NSE are closed by suitable models or even expressed analytically. The PC-IBM is fully compatible with the large eddy simulation framework, as the volume-filtered NSE converge to the filtered NSE away from solid boundaries. The potential of the PC-IBM is demonstrated by means of different particle-laden flow applications, including a dense packing of fixed particles and periodic settling of 500 particles. The new methodology turns out to be capable of accurately predicting the fluid forces on particles with relatively coarse mesh resolutions. For the flow around isolated spheres, a spatial resolution of six fluid mesh cells per diameter is sufficient to predict the drag force with less than 10% deviation from highly resolved simulation for the investigated particle Reynolds numbers ranging from 10 to 250.