Background: Nivolumab, a programmed death-1 antibody, is an immune checkpoint inhibitor approved in Japan in March 2017 for the treatment of recurrent or metastatic head and neck cancers (RM-HNCs) after platinum drug administration. This study aimed to evaluate the effectiveness and safety of nivolumab and to determine the prognostic factors affecting the treatment outcome, in a real-world setting in Japanese RM-HNCs. Methods: Forty-six patients with RM-HNCs treated with nivolumab between April 2017 and April 2021 at Shinshu University Hospital were retrospectively assessed in this cohort study. Results: The overall response rate was 17.4%, and the disease control rate was 41.3%. The median first and second progression-free survival (PFS1 and PFS2) were 2.6 and 10.3 months, respectively. The median overall survival (OS) was 14.8 months. Multivariate analysis showed that performance status (PS) (p = 0.003) and a decrease in neutrophil-lymphocyte ratio (NLR) (p = 0.02) were significantly associated with a better OS, and a decrease in NLR (p = 0.035) was associated with a better PFS2. Conclusions: This study is the first report of PFS2 in RM-HNCs treated with nivolumab; the long PFS2 may contribute to prolonged OS. We propose that the PS and a decrease in NLR could be useful clinical prognostic markers of nivolumab therapy, which can easily be evaluated in the clinical setting.