Western studies have shown a non-linear association between fish consumption and mortality, which might be explained by exposure to chemical contaminants. This study aims to explore the associations between fish consumption or omega-3 polyunsaturated fatty acids (n-3 PUFA) and mortality within the prospective E3N French cohort, and to investigate the role of dietary exposure to contaminants in these associations. In the E3N cohort composed of 72,585 women, we assessed fish consumption and n-3 PUFA intake through a food questionnaire sent in 1993. To estimate the dietary exposure to contaminants, we used the food contamination database of the second French total diet study. Cox proportional hazard models were used to estimate the association between fish, lean fish, fatty fish, and n-3 PUFA intake, with the risk of all-cause or cause-specific mortality. During the follow-up (1993–2014), 6,441 deaths were recorded. A U-shaped association was observed between fish consumption and all-cause mortality (Poverall_association = 0.017). A similar association was observed with lean fish consumption, while the non-linear association between fatty fish consumption or n-3 PUFA intake and all-cause mortality did not reach statistical significance. A non-linear association was observed between fish consumption and lung cancer mortality (Poverall_association = 0.005). A positive and linear association was observed between fatty fish consumption or n-3 PUFA intake and breast cancer mortality (HR [CI95%]: 1.07 [1.01–1.15] and 1.08 [1.01–1.15]). Our results remained unchanged when further adjusting on dietary exposure to contaminants. Our results showed a U-shaped association between fish consumption and all-cause mortality and suggest a notable role of lean fish consumption in this association, but no role of dietary exposure to contaminants. Further studies are needed to better clarify this U-shaped association and the different impacts of fatty and lean fish consumption on health.