Most elderly persons suffer from chronic diseases. To treat chronic diseases, good medication adherence and refilling medication as physicians’ advice are key points of pharmaceutical therapy. However, the literature revealed that only limited studies have focused on the medication adherence of the elderly in rural communities. This study evaluated the adherence to medication and the degree of refilling prescriptions as physician’s advice by the Adherence to Refills and Medications Scale (ARMS) and evaluated the factors associated with poor adherence to medication. This cross‐sectional study focused on the medication use of the elderly population in rural areas in western Taiwan and analyzed related personal and social factors which influence the behaviors of pharmaceutical therapy. In conclusion, most of the elderly who lived in rural areas in western Taiwan have poor adherence (ARMS score over 12) to medication and a poor degree of refilling prescriptions as per physicians’ advice. Moreover, age ≥75 years and low urbanization were significantly associated with poor adherence. Future research is warranted to assess the effect of the interventions to improve medication adherence on chronic disease control, disability, and mortality.