We use the ultra-deep HUDF09 and the deep ERS data from the HST WFC3/IR camera, along with the wide area Spitzer/IRAC data from GOODS-S to derive SEDs of star-forming galaxies from the rest-frame UV to the optical over a wide luminosity range (M_1500 ~ -21 to M_1500 ~ -18) from z ~ 7 to z ~ 4. The sample contains ~ 400 z ~ 4, ~ 120 z ~ 5, ~ 60 z ~ 6, and 36 prior z ~ 7 galaxies. Median stacking enables the first comprehensive study of very faint high-z galaxies at multiple redshifts (e.g., [3.6] = 27.4 +/- 0.1 AB mag for the M_1500 ~ -18 sources at z ~ 4). At z ~ 4 our faint median-stacked SEDs reach to ~ 0.06 L*(z=4) and are combined with recently published results at high luminosity L > L* that extend to M_1500 ~ -23. We use the observed SEDs and template fits to derive rest frame UV-to-optical colors (U - V) at all redshifts and luminosities. We find that this color does not vary significantly with redshift at a fixed luminosity. The UV-to-optical color does show a weak trend with luminosity, becoming redder at higher luminosities. This is most likely due to dust. At z >~ 5 we find blue colors [3.6]-[4.5] ~ -0.3 mag that are most likely due to rest-frame optical emission lines contributing to the flux in the IRAC filter bandpasses. The scatter across our derived SEDs remains substantial, but the results are most consistent with a lack of any evolution in the SEDs with redshift at a given luminosity. The similarity of the SEDs suggests a self-similar mode of evolution over a timespan from 0.7 Gyr to 1.5 Gyr that encompasses very substantial growth in the stellar mass density in the universe (from ~ 4x10^6 to ~ 2x10^7 Msun Mpc^-3).