Effects of socioeconomic factors and cancer survivors' worries on their quality of life (QOL) were investigated. In 2002, Japanese national survey was performed to assess distress among cancer patients using a semi-structured questionnaire (http://www.scchr.jp/yorozu/pdf/taiken_koe_eng.pdf). We investigated relationships between patients' distress and their QOL measured by European Organization for Research and Treatment of Cancer Core Questionnaire (EORTC QLQ-C30) and Functional Assessment of Chronic Illness Therapy--12-item Spiritual Well-Being Scale (FACIT-Sp), using a covariance structure analysis and multivariate regression analysis. A total of 130 outpatients (male: 42%; average age: 59 years; performance status rating 0-2:89%; breast/lung/gastrointestinal cancer: 38/22/21%) answered the questionnaires. A covariance structure analysis extracted latent variables, which were named socioeconomic distress and cancer worries, using a model that sufficiently represented the observed data (Goodness of fit index = 0.833). Regression analysis demonstrated that higher family income significantly correlated with better Global health status/QOL (p = 0.003) but that losing a job negatively correlated with all of the scales on functioning in the QLQ-C30 (p < 0.05) and spiritual well-being (p < 0.05). Patients' QOL was also affected by physical worries and spiritual issues in terms of emotional, cognitive, and social functioning. In conclusion, cancer survivors' QOL was doubly affected by socioeconomic distress and cancer worries. In the former, lower family income and losing employment by experiencing cancer had a negative impact on patients' QOL. As to the latter, physical worries and spiritual issues also affected patients' QOL.