Nearby dwarf galaxies display a variety of effective radii (sizes) at a given stellar mass, suggesting different evolution scenarios according to their final `stellar' size. The TNG hydrodynamical simulations present a bimodality in the $z=0$ size--mass relation (SMRz0) of dwarf galaxies, at $r_ star 450$ pc. Using the TNG50 simulation, we explored the evolution of the most massive progenitors of dwarf galaxies ($z=0$ lmsun between 8.4 and 9.2) that end up as central galaxies of their groups. We split these dwarfs into three classes of the SMRz0: `` '' from the central spine of the main branch, and `` '' from the secondary branch as well as the lower envelope of the main branch. Both classes of see their stellar sizes decrease from $z onwards in contrast to while the sizes of the gas and dark matter (DM) components continue to increase (as for A detailed analysis reveals that live in poorer environments, and thus suffer fewer major mergers from $z=0.8$ onwards, which otherwise would pump angular momentum into the gas, allowing strong gas inflows, producing inner star formation, and thus leading to the buildup of a stellar core. are predicted to be rounder and to have bluer cores. Compact dwarfs of similar sizes are observed in the GAMA survey, but the bimodality in size is less evident and the most compact dwarfs tend to be passive rather than star forming, as in TNG50. Our conclusions should therefore be confirmed with future cosmological hydrodynamical simulations.