Nearby dwarf galaxies display a variety of effective radii (sizes) at a given stellar mass, suggesting different evolution scenarios according to their final “stellar” size. The TNG hydrodynamical simulations present a bimodality in the z = 0 size–mass relation (SMRz0) of dwarf galaxies, at r1/2, ⋆ ∼ 450 pc. Using the TNG50 simulation, we explored the evolution of the most massive progenitors of dwarf galaxies (z = 0 log(M⋆/M⊙) between 8.4 and 9.2) that end up as central galaxies of their groups. We split these dwarfs into three classes of the SMRz0: “Normals” from the central spine of the main branch, and “Compacts” from the secondary branch as well as the lower envelope of the main branch. Both classes of Compacts see their stellar sizes decrease from z ∼ 1 onwards in contrast to Normals, while the sizes of the gas and dark matter (DM) components continue to increase (as for Normals). A detailed analysis reveals that Compacts live in poorer environments, and thus suffer fewer major mergers from z = 0.8 onwards, which otherwise would pump angular momentum into the gas, allowing strong gas inflows, producing inner star formation, and thus leading to the buildup of a stellar core. Compacts are predicted to be rounder and to have bluer cores. Compact dwarfs of similar sizes are observed in the GAMA survey, but the bimodality in size is less evident and the most compact dwarfs tend to be passive rather than star forming, as in TNG50. Our conclusions should therefore be confirmed with future cosmological hydrodynamical simulations.