Pemphigus vulgaris (PV) is a rare autoimmune bullous disease, which can present with recalcitrant oral mucosal lesions. Optimal management of PV relies upon careful clinical assessment and documentation. The primary aim of this study was to validate the Oral Disease Severity Score (ODSS) for the assessment of oral involvement in PV. A secondary aim was to compare its inter- and intraobserver variability and ease of use with the Physician's Global Assessment (PGA) and the oral scoring methods used in the Autoimmune Bullous Skin Disorder Intensity Score (ABSIS) and the Pemphigus Disease Area Index (PDAI). Fifteen patients with mild-to-moderately severe oral PV were scored for disease severity by 10 oral medicine clinicians using the ODSS, the PGA and the oral sections of ABSIS and PDAI. Two clinicians rescored all patients after a minimum 2-h interval. Interobserver reliability was assessed using an intraclass correlation coefficient (ICC). For the ODSS total score the ICC was 0·83, for PDAI (oral total activity) 0·79, ABSIS (oral total) 0·71 and PGA 0·7. Intraobserver agreement between initial scoring and rescoring of the same patient by two clinicians demonstrated an ICC for each of 0·97 and 0·96 for ODSS total score; 0·99 and 0·82 for PDAI oral activity; 0·86 and 0·45 for ABSIS total; and 0·99 and 0·64 for PGA. Convergent validity was good, with a correlation coefficient > 0·5 (P < 0·001). The mean ± SD times taken to complete each scoring method were ODSS 76 ± 37 s, PDAI 117 ± 16 s and ABSIS 75 ± 19 s. This study has validated the ODSS for the assessment of oral PV. It has shown superior inter- and intraobserver reliability to PDAI, ABSIS and PGA and is quick to perform.