Objective: Intake of n-3 polyunsaturated fatty acids (n-3 PUFAs) may protect against mild cognitive impairment (MCI). However, there is still a lack of the n-3 PUFAs intervention in the elderly with MCI in China. The aim of the present study was to investigate the effect of n-3 PUFA supplementation on cognitive function in the Chinese elderly with MCI. Methods: Eighty six MCI individuals aged 60 years or older were randomly assigned to receive either n-3 PUFAs (480 mg DHA and 720 mg EPA per day, n = 44) or placebo (olive oil, n = 42) capsules. The changes of cognitive functions were assessed using Basic Cognitive Aptitude Tests (BCAT). Results: The mean age of participants was 71 years old, and 59% of the participants were men. n-3 PUFA supplementation was associated with improved total BCAT scores, perceptual speed, space imagery efficiency, and working memory (p < 0.01), but not with mental arithmetic efficiency or recognition memory (p > 0.05). Subgroup analysis by sex showed that n-3 PUFAs significantly improved perceptual speed (p = 0.001), space imagery efficiency (p = 0.013), working memory (p = 0.018), and total BCAT scores (p = 0.000) in males. However, in females, the significant beneficial effects can only be observed in perceptual speed (p = 0.027), space imagery efficiency (p = 0.006), and total BCAT scores (p = 0.015)—not working memory (p = 0.113). Conclusion: n-3 PUFAs can improve cognitive function in people with MCI. Further studies with different fish oil dosages, longer intervention periods, and larger sample sizes should be investigated before definite recommendations can be made.