We conduct one of the largest systematic investigations of bright X-ray binaries (XRBs) in both young star clusters and ancient globular clusters (GCs) using a sample of six nearby star-forming galaxies. Combining complete CXO X-ray source catalogs with optical Physics at High Angular Resolution in Nearby Galaxies-Hubble Space Telescope cluster catalogs, we identify a population of 33 XRBs within or near their parent clusters. We find that GCs that host XRBs in spiral galaxies appear to be brighter, more compact, denser, and more massive than the general GC population. However, these XRB hosts do not appear to be preferentially redder or more metal-rich, pointing to a possible absence of the metallicity-boosted formation of low-mass X-ray binaries (LMXBs) that is observed in the GCs of older galaxies. We also find that a smaller fraction of LMXBs is found in spiral GC systems when compared with those in early-type galaxies: between 8% and 50%, or an average of 20% across galaxies in our sample. Although there is a non-negligible probability of a chance superposition between an XRB and an unrelated young cluster, we find that among clusters younger than 10 Myr, which most likely host high-mass XRBs, the fraction of clusters associated with an XRB increases at higher cluster masses and densities. The X-ray luminosity of XRBs appears to increase with the mass of the cluster host for clusters younger than ∼400 Myr, while the inverse relation is found for XRBs in GCs.