Modifying dietary behaviors into healthier habits may attenuate the risk of colorectal cancer (CRC) risk. This study aimed to investigate the association between dietary changes and the risk of CRC. Following dietary recommendations for red and processed meat, fruit and vegetables, and alcohol consumption, we classified 50,640 participants into poor and good adherence groups in the UK Biobank. Changes in dietary habits were defined as stable poor, poor to good, good to poor, and stable to good adherence. A Cox proportional hazard model was used to examine the association between dietary changes and CRC risk. Women were more likely to follow dietary recommendations than men. After a median of 3.3 years from the latest follow-up, 8,328 (16.4%) participants followed an improved dietary habit and 5,808 (11.5%) participants had a worsened diet. Compared to men who stably consumed fruit and vegetables <5 servings/day, those who increased their consumption to ≥5 servings/day were related to CRC risk reduction (hazard ratio: 0.24, [0.09-0.63]). However, the beneficial associations of increased fruit and vegetable consumption were not statistically significant in women (hazard ratio: 0.41, [0.11-1.56]). Our findings support the evidence that increasing fruit and vegetable intake could serve as a beneficial strategy to mitigate CRC risk in men. Participants from the UK Biobank significantly changes their adherence to dietary recommendations during the follow-up. Increasing fruit and vegetable consumption was inversely associated with CRC risk among men.