Optimal use of psychotropic medications for people living with dementia is important. By finding potentially modifiable factors, dementia care teams may find solutions to achieve the appropriate use of psychotropic drugs. This study aimed to elucidate patient and caregiver factors associated with the use of psychotropic drugs listed in the potentially inappropriate medications (PIMs) in community-dwelling people with dementia. This cross-sectional study enrolled 808 patients newly diagnosed with dementia, and their caregivers, from a dementia clinic at Changhua Christian Hospital. Patient and caregiver characteristics, care mode, and social resource usage were recorded. Multivariate logistic regression was used to identify factors associated with prescribing psychotropic medications. Of all the participants, 39.1% used at least one of psychotropic medication categorized as PIM. Patients with frontotemporal dementia, with behavior or psychological symptoms, or cared by sole foreign care workers; caregivers with higher depression scores, employed or non-spouse caregivers carried a higher risk of prescription of psychotropic medications listed in PIMs. Psychotropic drug prescriptions are associated with patient and caregiver factors. Therefore, implementing appropriate interventions, especially those targeting potentially modifiable factors, is important to reduce psychotropic medication use.