Context. The XMM-XXL survey has used observations from the XMM-Newton observatory to detect clusters of galaxies over a wide range in mass and redshift. The moderate PSF (FWHM ~ 6″ on-axis) of XMM-Newton means that point sources within or projected onto a cluster may not be separated from the cluster emission, leading to enhanced luminosities and affecting the selection function of the cluster survey. Aims. We present the results of short Chandra observations of 21 galaxy clusters and cluster candidates at redshifts z > 1 detected in the XMM-XXL survey in X-rays or selected in the optical and infra-red. Methods. With the superior angular resolution of Chandra, we investigate whether there are any point sources within the cluster region that were not detected by the XMM-XXL analysis pipeline, and whether any point sources were misclassified as distant clusters. Results. Of the 14 X-ray selected clusters, 9 are free from significant point source contamination, either having no previously unresolved sources detected by Chandra or with less than about 10% of the reported XXL cluster flux being resolved into point sources. Of the other five sources, one is significantly contaminated by previously unresolved AGN, and four appear to be AGN misclassified as clusters. All but one of these cases are in the subset of less secure X-ray selected cluster detections and the false positive rate is consistent with that expected from the XXL selection function modelling. We also considered a further seven optically selected cluster candidates associated with faint XXL sources that were not classed as clusters. Of these, three were shown to be AGN by Chandra, one is a cluster whose XXL survey flux was highly contaminated by unresolved AGN, while three appear to be uncontaminated clusters. By decontaminating and vetting these distant clusters, we provide a pure sample of clusters at redshift z > 1 for deeper follow-up observations, and demonstrate the utility of using Chandra snapshots to test for AGN in surveys with high sensitivity but poor angular resolution.