We exploit deep combined observations with Spitzer and Chandra of the Spitzer Wide-Area Infrared Extragalactic Survey (SWIRE) in the ELAIS N1 region to investigate the nature of the faint X-ray and IR sources in common, to identify active galactic nucleus (AGN)/starburst diagnostics, and to study the sources of the X-ray and IR cosmic backgrounds (XRB and CIRB). In the 17' × 17' area of the Chandra ACIS-I image there are approximately 3400 SWIRE near-IR sources with 4 σ detections in at least two Infrared Array Camera (IRAC) bands and 988 sources detected at 24 μm with the Multiband Imaging Photometer (MIPS) brighter than S24 sime 0.1 mJy. Of these, 102 IRAC and 59 MIPS sources have Chandra counterparts, out of a total of 122 X-ray sources present in the area with S0.5–8 keV > 10-15 ergs cm-2 s-1. We have constructed spectral energy distributions (SEDs) for each source using data from the four IRAC wavebands, Chandra fluxes in the hard (2–8 keV) and soft (0.5–2 keV) X-rays, and optical follow-up data in the wavebands U, g', r', i', Z, and H. We fit a number of spectral templates to the SEDs at optical and IR wavelengths to determine photometric redshifts and spectral categories and also make use of diagnostics based on the X-ray luminosities, hardness ratios, X-ray to IR spectral slopes, and optical morphologies. Although we have spectroscopic redshifts for only a minority of the Chandra sources (10 type 1 QSOs or Seyfert sources and three galaxies), the available SEDs constrain the redshifts for most of the sample sources, which turn out to be typically at 0.5 < z < 2. We find that 39% of the Chandra sources are dominated by type 1 AGN emission (QSOs or Seyfert 1), 23% display optical/IR spectra typical of type 2 AGNs, while the remaining 38% show starburst-like or even normal galaxy spectra (including five passively evolving early-type galaxies). Since we prove that all these galaxies are dominated by AGN emission in X-rays (considering their large 0.5–8 keV rest-frame X-ray luminosities and their high X-ray to IR flux ratios), this brings the fraction of type 1 AGNs to 80% of the type 2 AGNs; even assuming that all the Chandra sources undetected by Spitzer are type 2 AGNs, the type 1 fraction would exceed 1/3 of the total population. Our analysis of the mid-IR MIPS 24 μm–selected sources, making up ~50% of the CIRB, shows that the fraction of those dominated by an AGN (either type 1 or type 2) is relatively constant with the IR flux and around 10%–15%. Our combined IR and hard X-ray observations allow us to verify that the dust covering fraction in type 1 AGNs is widely distributed between ~10% and 100%. A significant fraction, from 15% to 30% or more, of the sources of the XRB are hosted in galaxies whose optical/IR spectra are dominated by starburst (or normal galaxy) emission and for which only the hard X-ray spectra reveal the presence of a moderately luminous hidden AGN.