Phosphorylated histone H2AX, termed ‘γH2AX’, mediates the chromatin response to DNA double strand breaks (DSBs) in mammalian cells. H2AX deficiency increases the numbers of unrepaired DSBs and translocations, which are partly associated with defects in non-homologous end joining (NHEJ) and contributing to genomic instability in cancer. However, the role of γH2AX in NHEJ of general DSBs has yet to be clearly defined. Here, we showed that despite little effect on overall NHEJ efficiency, H2AX deficiency causes a surprising bias towards accurate NHEJ and shorter deletions in NHEJ products. By analyzing CRISPR/Cas9-induced NHEJ and by using a new reporter for mutagenic NHEJ, we found that γH2AX, along with its interacting protein MDC1, is required for efficient classical NHEJ (C-NHEJ) but with short deletions and insertions. Epistasis analysis revealed that ataxia telangiectasia mutated (ATM) and the chromatin remodeling complex Tip60/TRRAP/P400 are essential for this H2AX function. Taken together, these data suggest that a subset of DSBs may require γH2AX-mediated short-range nucleosome repositioning around the breaks to facilitate C-NHEJ with loss of a few extra nucleotides at NHEJ junctions. This may prevent outcomes such as non-repair and translocations, which are generally more destabilizing to genomes than short deletions and insertions from local NHEJ.