Phenotypic variability within families with epilepsy is often observed, even when relatives share the same monogenic cause. We aimed to investigate whether common polygenic risk for epilepsy could explain the penetrance and phenotypic expression of rare pathogenic variants in familial epilepsies. We studied 58 clinically heterogeneous families with genetic epilepsy with febrile seizures plus (GEFS+). Relatives were coded as either unaffected or affected with epilepsy, and graded according to phenotype severity: no seizures, febrile seizures (FS) only, febrile seizures plus (FS+), generalised/focal epilepsy, or developmental and epileptic encephalopathy (DEE). Epilepsy polygenic risk scores (PRSs) were tested for association with epilepsy phenotype. Within families, the mean PRS difference was compared between pairs concordant versus discordant for phenotype severity. Statistical analyses were performed using mixed-effect regression models. 304 individuals segregating a known, or presumed, rare variant of large effect, were studied. Within families, higher epilepsy polygenic risk was associated with an epilepsy diagnosis (OR=1.39, 95% CI 1.08, 1.80, padj=0.040). Relatives with a more severe phenotype had a mean pairwise PRS difference of+0.19 higher than relatives with a milder phenotype (padj=0.010). The difference increased with greater phenotype discordance between relatives. As the cohort included two rare variants with >30 relatives each, variant-specific genotype-phenotype associations could also be analysed. Whilst the epilepsy PRS effect was strong for relatives segregating the GABRG2 p.Arg82Gln pathogenic variant (padj=0.0010), the effect was not significant for SCN1B p.Cys121Trp. We provide support for genetic background modifying the penetrance and phenotypic expression of rare variants associated with 'monogenic' epilepsies. In GEFS+families, relatives with higher epilepsy PRSs were more likely to show penetrance (epilepsy diagnosis) and a more severe phenotype. Variant-specific analyses suggest that some rare variants may be more susceptible to PRS modification, carrying important genetic counselling and disease prognostication implications for patients. National Health and Medical Research Council of Australia, Medical Research Future Fund of Australia.