Over the last few years, there has been increasing interest in the use of amorphous carbon thin films with low secondary electron yield (SEY) to mitigate electron multipacting in particle accelerators and RF devices. Previous works found that the SEY increases with the amount of incorporated hydrogen and correlates with the Tauc gap. In this work, we analyse films produced by magnetron sputtering with different contents of hydrogen and deuterium incorporated via the target poisoning and sputtering of CxDy molecules. XPS was implemented to estimate the phase composition of the films. The maximal SEY was found to decrease linearly with the fraction of the graphitic phase in the films. These results are supported by Raman scattering and UPS measurements. The graphitic phase decreases almost linearly for hydrogen and deuterium concentrations between 12% and 46% (at.), but abruptly decreases when the concentration reaches 53%. This vanishing of the graphitic phase is accompanied by a strong increase of SEY and the Tauc gap. These results suggest that the SEY is not dictated directly by the concentration of H/D, but by the fraction of the graphitic phase in the film. The results are supported by an original model used to calculate the SEY of films consisting of a mixture of graphitic and polymeric phases.