Determining the sources of solar brightness variations 1,2 , often referred to as solar noise 3 , is important because solar noise limits the detection of solar oscillations 3 , is one of the drivers of the Earth’s climate system 4,5 and is a prototype of stellar variability 6,7 —an important limiting factor for the detection of extrasolar planets. Here, we model the magnetic contribution to solar brightness variability using high-cadence 8,9 observations from the Solar Dynamics Observatory (SDO) and the Spectral And Total Irradiance REconstruction (SATIRE) 10,11 model. The brightness variations caused by the constantly evolving cellular granulation pattern on the solar surface were computed with the Max Planck Institute for Solar System Research (MPS)/University of Chicago Radiative Magnetohydrodynamics (MURaM) 12 code. We found that the surface magnetic field and granulation can together precisely explain solar noise (that is, solar variability excluding oscillations) on timescales from minutes to decades, accounting for all timescales that have so far been resolved or covered by irradiance measurements. We demonstrate that no other sources of variability are required to explain the data. Recent measurements of Sun-like stars by the COnvection ROtation and planetary Transits (CoRoT) 13 and Kepler 14 missions uncovered brightness variations similar to that of the Sun, but with a much wider variety of patterns 15 . Our finding that solar brightness variations can be replicated in detail with just two well-known sources will greatly simplify future modelling of existing CoRoT and Kepler as well as anticipated Transiting Exoplanet Survey Satellite 16 and PLAnetary Transits and Oscillations of stars (PLATO) 17 data.