Fillet color of Atlantic salmon plays a large role in consumer purchasing decisions; however, color can be influenced by a variety of factors, including diet and genetics. In a breeding program, selection for color fillets relies on accurate measurements taken during harvest evaluations. The harvest of fish can take place over several days to several weeks, with a large variation in the amount of time taken to process a single fish which could impact the recorded value of fillet color, ultimately affecting estimated breeding values (EBVs) for genetic selection. Therefore, we evaluated the impact of the 1) day of harvest, 2) hour of the day, and 3) duration between entering the processing facility and having color assessed (using SalmoFan and Minolta Chroma Meter, i.e., redness or a*, yellowness or b*, and lightness or L*) on the recorded fillet color of North American Atlantic salmon (Salmo salar) and on the EBVs. A total of 10,380 animals with color measurements were included from eight year classes (2011 to 2019). The color measurements (Minolta Chroma Meter and SalmoFan) took place over an average of three days. The phenotypes of animals were fitted with linear animal mixed models in ASReml-R to determine the magnitude of time related effects on color traits, of which there was a significant impact on color depending on duration, as well as the day measurements were taken (p-value <0.05). Generally, longer harvest days, as the duration to process fish increased, the L* also increased, however, the opposite was observed for shorter harvest days. There was also a significant effect of hour of the day on SalmoFan scores in some year classes (up to a − 1.75 score difference over a 24-h period), where as the hour increased, lower SalmoFan measurements were perceived, possibly due to eye fatigue or other human-operator biases. There were moderate to high heritabilities ranging from 0.298 (L* in the 2018 year class) to 0.864 (a* in the 2014 year class). When days of harvest, hour of the day and the duration were not accounted for in the models, heritabilities for color measurements were generally lower than the models with these time effects. The L* and SalmoFan showed the largest changes in most year classes. The EBVs of color measurements for each individual changed between models where these fixed time effects were included or not, indicating that these effects have the potential to bias breeding decisions. Therefore, it will be essential to consider the duration, day, and hour of measurement when evaluating the phenotypes of individuals for unbiased breeding decisions.