The number of gastrointestinal (GI) cancer survivors has been steadily increasing owing to early diagnosis and improved cancer treatment outcomes. The quality of life (QoL) of cancer survivors can provide distinct prognostic information and represent their functioning. This study aimed to investigate the levels of symptoms, psychological distress, and QoL of GI cancer survivors, and identify factors associated with QoL. A cross-sectional survey was conducted among 145 survivors of gastric or colorectal cancer in a university-affiliated hospital, Seoul, South Korea. The questionnaire consisted of the M. D. Anderson Symptom Inventory Gastrointestinal Cancer Module, Distress Thermometer, and brief version of the World Health Organization Quality of Life Assessment Instrument. Quantile regression was used to assess the associated factors of QoL. The 10th, 25th, 50th, 75th, and 90th conditional quantiles were considered. The most common symptoms were fatigue (24.9%), numbness or tingling (17.2%), feeling bloated (17.2%), dry mouth (15.9%), and difficulty remembering (11.8%). Thirty-two percent (47/145) of the participants reported severe distress. A level of symptoms was significantly associated in the 10th and 25th quantiles, representing poor QoL. Economic burden was a significant influencing factor in all quantiles. Our results indicate that high burden from symptoms might be associated with lower QoL in GI cancer survivors, and higher economic burden from cancer treatment was associated with lower QoL. These results suggest that symptom management and support for economic difficulties should be included in the strategies to enhance the QoL of GI cancer survivors.