Context. The source population(s) of ultra-high energy cosmic rays (UHECRs) with energies of ≥57 × 1018 eV (57 EeV) are still unknown and continue to be a subject of debate. The published arrival directions and energies of 303 UHECRs from the Pierre Auger Observatory and Telescope Array enable correlations with astronomical sources. However, deflections of UHECRs in the Galactic Magnetic Field (GMF) can be considerable, especially for heavy UHECR compositions. Aims. We aim to constrain the source(s) of UHECRs by focusing on the subset of UHECRs that experience minimal deflections by the Galactic magnetic field (GMF). Methods. We used Monte Carlo simulations of UHECR trajectories in a total of eight posited GMF models (and six UHECR compositions from H to Fe) to identify two subsamples of about 40 UHECRs. The H compositions of these subsamples suffer deflections of less than a few degrees in one (or most) of the GMFs. Both the total UHECR sample and the “least-deflected” (vs. the remaining “more-deflected”) subsamples of UHECRs were cross-correlated with catalogs of astronomical sources (at D ≲ 100 Mpc) and neutrino arrival directions. Results. For H compositions, the full sample of UHECRs is most closely correlated with Cen A, nearby (< 75 Mpc) radiogalaxies, and optically selected active galactic nuclei (AGNs); the least-deflected sub sample shows a most consistent correlation with nearby (D ≲ 25 Mpc) galaxies, although some GMF models show a preference for the IceCube muon-neutrino arrival detections. For oxygen compositions, the out-of-galaxy arrival directions of least-deflected UHECRs fall on the extragalactic plane (and thus close to nearby galaxies) in many GMFs. For Fe compositions, the out-of-galaxy UHECR arrival directions typically originate in a region ∼30 deg offset from Cen A, except in the case of the JF12 model; this offset can be decreased by re-scaling (by ∼50%) the magnitudes of the dipole and disk fields of the relevant GMFs.