The topography between the ocean and marginal sea basins is often complex with narrow channels and shallow sills. Two-layer exchange flows are usually observed in the channels with internal waves and small-scale instabilities at the interface. A high-resolution, non-hydrostatic model (NHWAVE) is used to simulate the interfacial mixing processes. Excellent agreement with the experiment data is obtained, especially for the simulation of vertical velocity and internal wave dispersion. In the mixing processes, turbulent coherent structures vary from spanwise uniform vortices to full 3D vortices with random shapes of spanwise and streamwise vortices. These vortices correspond to the surface velocity divergence with wave-like perturbations and the crests of the vortices associate with the down zero crossings of the divergence. The non-hydrostatic effect is also analyzed by an aspect ratio. The results show that the most obvious non-hydrostatic areas are located at the lee side of the sill, ranging from x = 0–1 m. And the effect of non-hydrostatic is more significant with larger values of the vertical velocity gradient.