ABSTRACT Angular power spectra of optical and infrared background anisotropies at wavelengths between 0.5 and 5 μm are a useful probe of faint sources present during reionization, in addition to faint galaxies and diffuse signals at low redshift. The cross-correlation of these fluctuations with backgrounds at other wavelengths can be used to separate some of these signals. A previous study on the cross-correlation between X-ray and Spitzer fluctuations at 3.6 μm and 4.5 μm has been interpreted as evidence for direct collapse black holes present at z > 12. Here we return to this cross-correlation and study its wavelength dependence from 0.5 to 4.5 μm using Hubble and Spitzer data in combination with a subset of the 4 Ms Chandra observations in GOODS-S/ECDFS. Our study involves five Hubble bands at 0.6, 0.7, 0.85, 1.25, and 1.6 μm, and two Spitzer-IRAC bands at 3.6 μm and 4.5 μm. We confirm the previously seen cross-correlation between 3.6 μm (4.5 μm) and X-rays with 3.7σ (4.2σ) and 2.7σ (3.7σ) detections in the soft [0.5–2] keV and hard [2–8] keV X-ray bands, respectively, at angular scales above 20 arcsec. The cross-correlation of X-rays with Hubble is largely anticorrelated, ranging between the levels of 1.4σ–3.5σ for all the Hubble and X-ray bands. This lack of correlation in the shorter optical/NIR bands implies the sources responsible for the cosmic infrared background at 3.6 and 4.5 μm are at least partly dissimilar to those at 1.6 μm and shorter.