We present a measurement of the surface brightness of the cosmic X-ray background (CXB) in the Chandra Deep Fields, after excluding all detected X-ray, optical and infrared sources. The work is motivated by a recent X-ray stacking analysis by Worsley and collaborators, which showed that galaxies detected by HST but not by Chandra may account for most of the unresolved CXB at E>1 keV. We find that after excluding HST and Spitzer IRAC sources, some CXB still remains, but it is marginally significant: (3.4+/-1.4)x10^-13 ergs cm^-2 s^-1 deg^2 in the 1-2 keV band and (4+/-9)x10^-13 ergs cm^-2 s^-1 deg^2 in the 2-5 keV band, or 7%+/-3% and 4%+/-9% of the total CXB, respectively. Of the 1-2 keV signal resolved by the HST sources, 34%+/-2% comes from objects with optical colors typical of ``normal'' galaxies (which make up 25% of the HST sources), while the remaining flux comes from objects with colors of starburst and irregular galaxies. In the 0.65-1 keV band (just above the bright Galactic O VII line) the remaining diffuse intensity is (1.0+/-0.2)x10^-12 ergs cm^-2 s^-1 deg^2. This flux includes emission from the Galaxy as well as from the hypothetical warm-hot intergalactic medium (WHIM), and provides a conservative upper limit on the WHIM signal that comes interestingly close to theoretical predictions.