This paper presents a mass and momentum conservative semi-implicit finite volume (FV) scheme for complex non-hydrostatic free surface flows, interacting with moving solid obstacles. A simplified incompressible Baer-Nunziato type model is considered for two-phase flows containing a liquid phase, a solid phase, and the surrounding void. According to the so-called diffuse interface approach, the different phases and consequently the void are described by means of a scalar volume fraction function for each phase. In our numerical scheme, the dynamics of the liquid phase and the motion of the solid are decoupled. The solid is assumed to be a moving rigid body, whose motion is prescribed. Only after the advection of the solid volume fraction, the dynamics of the liquid phase is considered. As usual in semi-implicit schemes, we employ staggered Cartesian control volumes and treat the nonlinear convective terms explicitly, while the pressure terms are treated implicitly. The non-conservative products arising in the transport equation for the solid volume fraction are treated by a path-conservative approach. The resulting semi-implicit FV discretization of the mass and momentum equations leads to a mildly nonlinear system for the pressure which can be efficiently solved with a nested Newton-type technique. The time step size is only limited by the velocities of the two phases contained in the domain, and not by the gravity wave speed nor by the stiff algebraic relaxation source term, which requires an implicit discretization. The resulting semi-implicit algorithm is first validated on a set of classical incompressible Navier-Stokes test problems and later also adds a fixed and moving solid phase.