So far, numerical simulations of ultra-faint dwarf galaxies (UFDs) have failed to properly reproduce the observed size–luminosity relation. In particular, no hydrodynamical simulation run has managed to form UFDs with a half-light radius as small as 30 pc, as seen in observations of several UFD candidates. We tackle this problem by developing a simple but numerically clean and powerful method in which predictions of the stellar content of UFDs from ΛCDM cosmological hydrodynamical simulations are combined with very high-resolution dark-matter-only runs. This method allows us to trace the buildup history of UFDs and to determine the impact of the merger of building-block objects on their final size. We find that, while no UFDs more compact than 20 pc can be formed, slightly larger systems are only reproduced if all member stars originate from the same initial mini-halo. However, this imposes that (i) the total virial mass is smaller than 3 × 108 M⊙, and (ii) the stellar content prior to the end of the reionisation epoch is very compact (< 15 pc) and strongly gravitationally bound, which is a challenge for current hydrodynamical numerical simulations. If initial stellar building blocks are larger than 35 pc, the size of the UFD will extend to 80 pc. Finally, our study shows that UFDs keep strong imprints of their buildup history in the form of elongated or extended stellar halos. Those features can erroneously be interpreted as tidal signatures.