We have studied the correlation between 2357 Chandra X-ray point sources in a 40 × 40 pc field and ~20,000 infrared sources we observed in the corresponding subset of our 2° × 1.4° Spitzer/IRAC Galactic Center Survey at 3.6-8.0 μm, using various spatial and X-ray hardness thresholds. The correlation was determined for source separations of less than 0.5'', 1'', or 2''. Only the soft X-ray sources show any correlation with infrared point sources on these scales, and that correlation is very weak. The upper limit on hard X-ray sources that have infrared counterparts is <1.7% (3 σ). However, because of the confusion limit of the IR catalog, we only detect IR sources with absolute magnitudes ≲1. As a result, a stronger correlation with fainter sources cannot be ruled out. Only one compact IR source, IRS 13, coincides with any of the dozen prominent X-ray emission features in the 3 × 3 pc region centered on Sgr A*, and the diffuse X-ray and IR emission around Sgr A* seems to be anticorrelated on a few-arcsecond scale. We compare our results with previous identifications of near-infrared companions to Chandra X-ray sources.