This longitudinal study examined psychological and social activity factors related to poor self-rated health (SRH) in community-dwelling elderly people. The general health of 7,413 elderly individuals aged 65 years and over in Toon City, Ehime Prefecture, Japan was surveyed. We followed 4,372 participants, over a five-year period, after excluding those who were aged 85 years and over, had a disability, had moved away, or had died. The data from 3,358 respondents (response rate: 76.8%) were analyzed. We divided the patients into two groups based on their SRH responses: healthy, including those who answered "excellent" or "good," and unhealthy, including those who answered "not good" or "poor." We examined changes in SRH for both groups between the first survey and the survey conducted after five years. Among the healthy subjects at the first survey, we analyzed the relationship between SRH, after five years, and psycho-social activity factors using a logistic regression analysis. These factors included physical and social competence, life satisfaction, and tendency towards dementia and/or depression. SRH of both men and women significantly declined over five years. The percentage of men and women, who maintained SRH as healthy, after the 5-year follow-up period, was approximately 60% in those aged 65-74 years and 40% in those aged 75-84 years. In those aged 65-74 years, the odds ratio (OR) for a SRH of unhealthy (after five years), associated with Life Satisfaction Index-K (LSI-K) scores (at the first survey), was significantly lower at 0.85 (95% confidence interval [CI]: 0.77-0.93) for men and 0.79 (95% CI: 0.72-0.87) for women. The OR of tendency toward depression was significantly higher at 1.68 (95% CI: 1.11-2.56) for women only. In those aged 75-84 years, the OR for a SRH of unhealthy (after five years), associated with LSI-K scores (at the first survey), was significantly lower at 0.87 (95% CI: 0.77-1.00) for men and 0.89 (95% CI: 0.80-0.99) for women. The OR for higher Tokyo Metropolitan Institute of Gerontology scores was significantly lower at 0.80 (95% CI: 0.70-0.91) for men and 0.88 (95% CI: 0.80-0.97) for women. This study shows that it is necessary to increase life satisfaction for preventing a decline in SRH in community-dwelling elderly. It also shows that preventing depression in elderly women under the age of 75 years and maintaining physical and social competence in both elderly men and women aged 75-84 years are important for maintaining the SRH status.