Understanding the evolution of the solar magnetic field is of great importance for heliosphere, dynamo, and irradiance studies, for example. While the contribution of the field in active regions (ARs) hosting sunspots to the Sun's large-scale field has been extensively modelled, we still lack a realistic model of the contribution of smaller-scale magnetic regions such as ephemeral regions which do not contain any sunspots. For this work, we studied the effect of small and large bipolar magnetic regions (BMRs) on the large-scale solar magnetic field. The evolution of the total and open magnetic flux, the polar fields, and the toroidal flux loss since 1874 has been simulated with a surface flux transport model (SFTM) and the results were compared to analytical considerations and observational data. For this purpose, we constructed semi-synthetic BMR records using the international sunspot number as a proxy. We calculated the emergence rate of all BMRs from a single power-law size distribution, whose exponent varies with solar activity. The spatial distribution of the BMRs was calculated from statistical relationships derived from various solar observations. We included BMRs with a magnetic flux as low as $2 $ Mx in our SFTM, corresponding to regions with lifetimes down to one day. We found a good agreement between the computed total magnetic flux and observations, even though we do not have a free parameter to adjust the simulated total flux to observations, as in earlier versions of the employed SFTM. The open flux, the polar fields, and the toroidal flux loss are also consistent with observations and independent reconstructions. In our model, small BMRs contribute about one-third of the total and open flux at activity maximum, while their contribution increases to roughly half at activity minimum. An even greater impact is found on the polar fields and the toroidal flux loss, for which the contribution of small BMRs is comparable to that of spot-containing ARs at all activity levels. Even so, smaller regions, not included in our simulations, do not seem to play a significant role due to their high tilt angle scatter. Our simulation results suggest that most of the statistical noise is caused by large ARs, while small BMRs have a stabilising effect on the magnetic flux evolution, especially for the polar field reversals. We conclude that small BMRs (here, with magnetic fluxes between $2 $ Mx and $3 $ Mx) may also play an important role in the evolution of the solar magnetic field at large spatial scales. Their impact is largest at low solar activity, but it is also substantial during activity maxima, although the actual relative contributions by small and large regions depend on the steepness of their emergence rate distribution. The inclusion of small BMRs in SFTM simulations will allow the secular variability in solar irradiance to be better constrained and the generation of the poloidal field in the Babcock-Leighton dynamo to be better understood.