Studies of the cognitive abilities of animals aim to help us understand how they communicate, obtain resources, avoid danger and otherwise thrive in a given environment. But to what extent is cognitive ability a fixed trait in individuals? And can we answer this question by measuring performance on tests of cognitive ability? We tested the same 18 male song sparrows, Melospiza melodia, once yearly, across three consecutive years, with tests of four putative cognitive traits and a test of neophobia. We also tested 19 females twice, once in the first year and once in the third year. All birds were hand-reared and tested in the laboratory. Analyses of both data sets indicate repeatability of neophobia but not of performance on the cognitive tests. In addition, correlations among cognitive performance, neophobia and song quality that were observed in the first year were not observed in subsequent rounds of testing. These results suggest that cognitive ability is not a fixed trait in individuals, or that the tests used do not accurately measure cognitive ability, or both. Conclusions drawn from a single round of cognitive tests should therefore be interpreted with caution in this species and in any species in which repeatability has not been verified.