An ocean circulation model for process studies of the Subpolar North Atlantic is developed based on the Geophysical Fluid Dynamics Laboratory (GFDL) Modular Ocean Model (MOM) code. The basic model configuration is identical with that of the high‐resolution model (with a grid size of 1/3° × 2/5°) of the World Ocean Circulation Experiment (WOCE) Community Modeling Effort (CME), except that the domain of integration is confined to the area from 43° to 65°N. Open boundary conditions are used for the inflows and outflows across the northern and southern boundaries. A comparison with the CME model covering the whole North Atlantic (from 15°S to 65°N) shows that the regional model, with inflow conditions at 43°N from a CME solution, is able to reproduce the CME results for the subpolar area. Thus the potential of a regional model lies in its use as an efficient tool for numerical experiments aiming at an identification of the key physical processes that determine the circulation and water mass transformations in the subpolar gyre. This study deals primarily with the representation and role of the overflow waters that enter the domain at the northern boundary. Sensitivity experiments show the effect of closed versus open boundaries, of different hydrographic conditions at inflow points, and of the representation of the narrow Faeroe Bank Channel. The representation of overflow processes in the Denmark Strait is the main controlling mechanism for the net transport of the deep boundary current along the Greenland continental slope and further downstream. Changes in the Faeroe Bank Channel throughflow conditions have a comparatively smaller effect on the deep transport in the western basin but strongly affect the water mass characteristics in the eastern North Atlantic. The deep water transport at Cape Farewell and further downstream is enhanced compared to the combined Denmark Strait and Iceland‐Scotland overflows. This enhancement can be attributed to a barotropic recirculation in the Irminger Basin which is very sensitive to the outflow conditions in the Denmark Strait. The representation of both overflow regions determine the upper layer circulation in the Irminger and Iceland Basins, in particular the path of the North Atlantic Current.