The role of the oral microbiome in mental health has recently been appreciated within the proposed oral-brain axis. This study examined the structure and composition of the salivary microbiome in a large-scale population-based cohort of individuals reporting mental health symptoms (n = 306) compared to mentally healthy controls (n = 164) using 16S rRNA sequencing. Mental health symptoms were evaluated using validated questionnaires and included depression, anxiety, and posttraumatic stress disorder (PTSD), with accompanying periodontal outcomes. Participants also indicated current or previous diagnoses of anxiety, depression, periodontitis, and gingivitis. Mental and periodontal health variables influenced the overall composition of the oral microbiome. PTSD symptoms correlated with a lower clr-transformed relative abundance of Haemophilus sputorum and a higher clr-transformed relative abundance of Prevotella histicola. The clr-transformed relative abundance of P. histicola was also positively associated with depressive scores and negatively associated with psychological quality of life. Anxiety disorder diagnosis was associated with a lower clr-transformed relative abundance of Neisseria elongate and a higher clr-transformed relative abundance of Oribacterium asaccharolyticum. A higher clr-transformed relative abundance of Shuttleworthia and lower clr-transformed relative abundance of Capnocytophaga were evident in those who reported a clinical periodontitis diagnosis. Higher Eggerthia and lower Haemophilus parainfluenzae clr-transformed relative abundances were associated with reported clinical periodontitis diagnoses and psychotherapeutic efficacy. Functional prediction analysis revealed a potential role for tryptophan metabolism/degradation in the oral-brain axis, which was confirmed by lower plasma serotonin levels across symptomatic groups. This study sheds light on the intricate interplay between oral microbiota, periodontal and mental health outcomes, and a potential role for tryptophan metabolism in the proposed oral-brain axis, emphasizing the need for further exploration to pave the way for novel therapeutic interventions and predicting therapeutic response.