Measuring the build-up of stellar mass is one of the main objectives of studies of galaxy evolution. Traditionally, the mass in stars and the star formation rates have been measured by different indicators, such as photometric colours, emission lines, and the UV and IR emission. We intend to show that it is possible to derive the physical parameters of galaxies from their broad-band spectral energy distribution out to a redshift of 1.2. This method has the potential to yield the physical parameters of all galaxies in a single field in a homogeneous way, thus overcoming problems with the sample size that particularly plague methods relying on spectroscopy. We use an extensive dataset, assembled in the context of the VVDS survey, which reaches from the UV to the IR and covers a sample of 84 073 galaxies over an area of 0.89 deg^2. We also use a library of 100 000 model galaxies with a wide variety of star formation histories (in particular including late bursts of star formation). We find that we can determine the physical parameters stellar mass, age, and star formation rate with good confidence. We validate the star formation rate determination in particular by comparing it to a sample of spectroscopically observed galaxies with an emission-line measurement. While the attenuation in the galaxies shows more scatter, the mean over the sample is unbiased. Metallicity, however, cannot be measured from rest-frame optical photometry alone. As a first application we use our sample to build the number density function of galaxies as a function of stellar mass, specific star formation rate, and redshift. We are then able to study whether the stellar mass function at a later time can be predicted from the stellar mass function and star formation rate distribution at an earlier time. We find that, between redshifts of 1.02 and 0.47, the predicted growth in stellar mass from star formation agrees with the observed one. However, the predicted stellar mass density for massive galaxies is lower than observed, while the mass density of intermediate mass galaxies is overpredicted. This apparent discrepancy can be explained by major and minor mergers. Indeed, when comparing with a direct measurement of the major merger rate from the VVDS survey, we find that major mergers can account for about half of the mass build-up at the massive end. Minor mergers are very likely to contribute the missing fraction. Based on data obtained with the European southern observatory Very Large Telescope, Paranal, Chile, programme 070A-9007(A) and on observations obtained with MegaPrime/MegaCam, a joint project of CFHT and CEA/DAPNIA, at the Canada-France-Hawaii Telescope (CFHT), which is operated by the National Research Council (NRC) of Canada, the Institut National des Science de l'Univers of the Centre National de la Recherche Scientifique (CNRS) of France, and the University of Hawaii. This work is based in part on data products produced at TERAPIX and the Canadian Astronomy Data Centre as part of the Canada-France-Hawaii Telescope Legacy Survey, a collaborative project of NRC and CNRS.