Genome-wide association studies have identified multiple Alzheimer's disease risk loci with small effect sizes. Polygenic risk scores, which aggregate these variants, are associated with grey matter structural changes. However, genome-wide scores do not allow mechanistic interpretations. The present study explored associations between disease pathway-specific scores and grey matter structure in younger and older adults. Data from two separate population cohorts were used as follows: the Avon Longitudinal Study of Parents and Children, mean age 19.8, and UK Biobank, mean age 64.4 (combined n = 18 689). Alzheimer's polygenic risk scores were computed using the largest genome-wide association study of clinically assessed Alzheimer's to date. Relationships between subcortical volumes and cortical thickness, pathway-specific scores and genome-wide scores were examined. Increased pathway-specific scores were associated with reduced cortical thickness in both the younger and older cohorts. For example, the reverse cholesterol transport pathway score showed evidence of association with lower left middle temporal cortex thickness in the younger Avon participants (P = 0.034; beta = -0.013, CI -0.025, -0.001) and in the older UK Biobank participants (P = 0.019; beta = -0.003, CI -0.005, -4.56 × 10-4). Pathway scores were associated with smaller subcortical volumes, such as smaller hippocampal volume, in UK Biobank older adults. There was also evidence of positive association between subcortical volumes in Avon younger adults. For example, the tau protein-binding pathway score was negatively associated with left hippocampal volume in UK Biobank (P = 8.35 × 10-05; beta = -11.392, CI -17.066, -5.718) and positively associated with hippocampal volume in the Avon study (P = 0.040; beta = 51.952, CI 2.445, 101.460). The immune response score had a distinct pattern of association, being only associated with reduced thickness in the right posterior cingulate in older and younger adults (P = 0.011; beta = -0.003, CI -0.005, -0.001 in UK Biobank; P = 0.034; beta = -0.016, CI -0.031, -0.001 in the Avon study). The immune response score was associated with smaller subcortical volumes in the older adults, but not younger adults. The disease pathway scores showed greater evidence of association with imaging phenotypes than the genome-wide score. This suggests that pathway-specific polygenic methods may allow progress towards a mechanistic understanding of structural changes linked to polygenic risk in pre-clinical Alzheimer's disease. Pathway-specific profiling could further define pathophysiology in individuals, moving towards precision medicine in Alzheimer's disease.