Agyrophilic grains (AGs) are age-related limbic-predominant lesions in which four-repeat tau is selectively accumulated. Because previous methodologically heterogeneous studies have demonstrated inconsistent findings on the relationship between AGs and dementia, whether AGs affect cognitive function remains unclear. To address this question, we first comprehensively evaluated the distribution and quantity of Gallyas-positive AGs and the severity of neuronal loss in the limbic, neocortical, and subcortical regions in 30 cases of pure argyrophilic grain disease (pAGD) in Braak stages I–IV and without other degenerative diseases, and 34 control cases that had only neurofibrillary tangles with Braak stages I–IV and no or minimal Aβ deposits. Then, we examined whether AGs have independent effects on neuronal loss and dementia by employing multivariate ordered logistic regression and binomial logistic regression. Of 30 pAGD cases, three were classified in diffuse form pAGD, which had evident neuronal loss not only in the limbic region but also in the neocortex and subcortical nuclei. In all 30 pAGD cases, neuronal loss developed first in the amygdala, followed by temporo-frontal cortex, hippocampal CA1, substantia nigra, and finally, the striatum and globus pallidus with the progression of Saito AG stage. In multivariate analyses of 30 pAGD and 34 control cases, the Saito AG stage affected neuronal loss in the amygdala, hippocampal CA1, temporo-frontal cortex, striatum, globus pallidus, and substantia nigra independent of the age, Braak stage, and limbic-predominant age-related TDP-43 encephalopathy (LATE-NC) stage. In multivariate analyses of 23 pAGD and 28 control cases that lacked two or more lacunae and/or one or more large infarctions, 100 or more AGs per × 400 visual field in the amygdala (OR 10.02, 95% CI 1.12–89.43) and hippocampal CA1 (OR 12.22, 95% CI 1.70–87.81), and the presence of AGs in the inferior temporal cortex (OR 8.18, 95% CI 1.03–65.13) affected dementia independent of age, moderate Braak stages (III–IV), and LATE-NC. Given these findings, the high density of limbic AGs and the increase of AGs in the inferior temporal gyrus may contribute to the occurrence of dementia through neuronal loss, at least in cases in a low to moderate Braak stage.