Aims. We study the stellar (i.e., rest-optical) and dust-obscured star-forming (i.e., rest-mid-infrared) morphologies (i.e., sizes and Sérsic indices) of star-forming galaxies (SFGs) at 0.1 < z < 2.5. Methods. We combined Hubble Space Telescope (HST) images from the Cosmic Assembly Near-infrared Deep Extragalactic Legacy Survey (CANDELS) with JWST images from the Cosmic Evolution Early Release Science (CEERS) survey to measure the stellar and dust-obscured star formation distributions of 69 SFGs. Rest-mid-infrared (rest-MIR) morphologies were determined using a Markov chain Monte Carlo (MCMC) approach applied to the sharpest Mid-InfraRed Instrument (MIRI) images (i.e., shortest wavelength) dominated by dust emission (Sνdust/Sνtotal > 75%), as inferred for each galaxy from our optical-to-far-infrared spectral energy distribution fits with CIGALE. Rest-MIR Sérsic indices were only measured for the brightest MIRI sources, that is, with a signal-to-noise (S/N) greater than 75 (35 galaxies). At a lower S/N, simulations do indeed show that simultaneous measurements of both the size and Sérsic index become less reliable. We extended our study to fainter sources (i.e., S/N > 10; 69 galaxies) by restricting our structural analysis to their rest-MIR sizes (ReMIR) and by fixing their Sérsic index to a value of one. Results. Our MIRI-selected sample corresponds to a mass-complete sample (> 80%) of SFGs down to stellar masses 109.5, 109.5, and 1010 M⊙ at z ∼ 0.3, 1, and 2, respectively. The rest-MIR Sérsic index of bright galaxies (S/N > 75) has a median value of 0.7−0.3+0.8 (the range corresponds to the 16th and 84th percentiles), which is in good agreement with their median rest-optical Sérsic indices. The Sérsic indices as well as the distribution of the axis ratio of these galaxies suggest that they have a disk-like morphology in the rest-MIR. Galaxies above the main sequence (MS) of star formation (i.e., starbursts) have rest-MIR sizes that are, on average, a factor ∼2 smaller than their rest-optical sizes (ReOpt.). The median rest-optical to rest-MIR size ratio of MS galaxies increases with their stellar mass, from 1.1−0.2+0.4 at ∼109.8 M⊙ to 1.6−0.3+1.0 at ∼1011 M⊙. This mass-dependent trend resembles the one found in the literature between the rest-optical and rest-near-infrared sizes of SFGs, suggesting that it is primarily due to radial color gradients affecting rest-optical sizes and that the sizes of the stellar and star-forming components of SFGs are, on average, consistent at all masses. There is, however, a small population of SFGs (∼15%) with a compact star-forming component embedded in a larger stellar structure, with ReOpt.c > 1.8 × ReMIR. This population could be the missing link between galaxies with an extended stellar component and those with a compact stellar component, the so-called blue nuggets.