The presence of extracapsular spread (ECS) in traditional head and neck cancer generally results in the use of adjuvant chemoradiation (CRT). The NCCN level of evidence for this recommendation is “Category 1”, ie, “based upon high-level evidence”. In the case of oropharynx cancer (OPC), this recommendation and the clinical research from which it is based do not reflect p16 status. Increasing number of observational studies report that p16+ OPC patients have more favorable outcomes and require less aggressive treatment than traditional OPC patients. We reviewed and critically appraised the research upon which the recommendation for adjuvant CRT is based and assessed its relevance in the p16+ era. A review and critical appraisal of the published articles for trials RTOG 9501 and EORTC 22931, the collaborative comparative analysis of both trials, and the long-term follow-up of RTOG 9501 was performed. Particular attention was given to the design, conduct, analysis, results reporting, and generalizability using standard methodological criteria. A variety of methodological problems were identified. The study populations included a variety of sub sites (oral cavity, oropharynx, larynx, hypopharynx) and in RTOG 9501 included a preponderance of good prognosis sites in the CRT arm. In neither trial was p16 status determined and overall severity of comorbidity reported. Multiple patients were excluded after randomization. Results were presented as relative and not absolute effects. Univariate and unplanned sub-group analyses presented for positive margin and/or ECS; effect of CRT on ECS alone not reported. Multivariable analyses, appropriate for heterogeneity in patient and tumor factors, not done. No patient-reported outcomes reported. Impact of sample size and possibility of spurious findings with sub-group analyses not acknowledged. This critical appraisal of the literature on the role of adjuvant CRT in OPC patients with ECS identified multiple problems. Flaws in trial conduct and analysis, along with other shortcomings, undermine internal validity. Results of the trials are not generalizable to newly diagnosed patients in the p16+ era. The literature upon which the guidelines for the use of adjuvant CRT are based is neither methodologically sound nor relevant to the current management of p16+ OPC patients with ECS.