Nowadays, the growing number of people aged 65+ has become a global phenomenon. At that age, the most common medical problems are multimorbidity and inappropriate polypharmacy, which have a negative impact on the quality of life in older adults. The aim of this cross-sectional study was to examine comorbidity, the use of psychopharmaceuticals, and symptoms of addiction to anxiolytics among older adults living at home or in retirement homes, and to examine the differences in quality of life in relation to the use and misuse of psychotropic drugs. The research included 383 people aged 65+ living in the Republic of Croatia (EU). A standardized questionnaire CAGE was used to collect data about the use of psychotropic drugs. Quality of life was examined using the WHOQOL-BREF scale. The average age of respondents was 83 years. There is a significantly higher prevalence of anxiety disorders (p = 0.001) in respondents who live at home. Psychopharmaceuticals were used by 218 (56.9%) respondents, equally in both groups of respondents. A total of 77 (20.1%) respondents had been using anxiolytics for more than five years, while 26 (6.8%) of them had significant clinical symptoms of addiction to anxiolytics. All domains and the overall quality of life scale were significantly lower (p < 0.001) in respondents who have clinical symptoms of anxiolytic addiction. The results indicate that the use of psychotropic drugs by respondents is inappropriate. Respondents who inappropriately and excessively use psychotropic drugs have a significantly worse quality of life.