While sex differences in the incidence and mortality of colorectal cancer (CRC) are well documented, less is known about sex differences in patients' health-related quality of life (HRQoL) and psychological distress. To enhance patient-tailored care, we aimed to longitudinally examine sex differences in HRQoL and psychological distress among CRC patients from diagnosis up until 2-year follow-up. Newly diagnosed CRC patients from four Dutch hospitals were eligible for participation. Patients (N = 334) completed questions on HRQoL (EORTC QLQ-C30) and psychological distress (HADS) before initial treatment (baseline), 4 weeks after surgery, and at 1 and 2 years after diagnosis. Also, HRQoL and psychological distress were assessed in a sex- and age-matched reference population. When directly comparing female (N = 126, 38%) and male (N = 208, 62%) CRC patients, female patients reported significantly worse HRQoL, such as more insomnia at baseline, worse physical and role functioning 4 weeks after surgery, more diarrhea at 1 year, and more pain and constipation at 2-year follow-up. However, a comparison with the reference population revealed larger differences between patients and reference in males than in females. For example, at 1- and 2-year follow-up, male patients reported significantly worse cognitive and social functioning, more insomnia, and more anxiety compared with a reference population. Especially male CRC patients reported worse HRQoL and more psychological distress when compared with a reference population. Knowledge of sex-specific differences in HRQoL and psychological distress among CRC patients may help healthcare providers anticipate and appropriately address patients' unique healthcare needs.