We present a spatially resolved study of stellar populations in six galaxies with stellar masses M * ∼ 1010 M ☉ at z ∼ 3.7 using 14-filter James Webb Space Telescope (JWST)/NIRCam imaging from the JADES and JEMS surveys. The six galaxies are visually selected to have clumpy substructures with distinct colors over rest frame 3600−4100 Å, including a red, dominant stellar core that is close to their stellar-light centroids. With 23-filter photometry from the Hubble Space Telescope to JWST, we measure the stellar-population properties of individual structural components via spectral energy distribution fitting using Prospector. We find that the central stellar cores are ≳2 times more massive than the Toomre mass, indicating they may not form via single in situ fragmentation. The stellar cores have stellar ages of 0.4−0.7 Gyr that are similar to the timescale of clump inward migration due to dynamical friction, suggesting that they likely instead formed through the coalescence of giant stellar clumps. While they have not yet quenched, the six galaxies are below the star-forming main sequence by 0.2−0.7 dex. Within each galaxy, we find that the specific star formation rate is lower in the central stellar core, and the stellar-mass surface density of the core is already similar to quenched galaxies of the same masses and redshifts. Meanwhile, the stellar ages of the cores are either comparable to or younger than the extended, smooth parts of the galaxies. Our findings are consistent with model predictions of the gas-rich compaction scenario for the buildup of galaxies’ central regions at high redshifts. We are likely witnessing the coeval formation of dense central cores, along with the onset of galaxy-wide quenching at z > 3.