Selective serotonin reuptake inhibitors (SSRIs) are a common first-line treatment for some psychiatric disorders, including depression and anxiety; although they are generally well tolerated, SSRIs have known adverse effects, including movement problems, sleep disruption, and gastrointestinal problems (eg, nausea and upset stomach). No large-scale studies using naturalistic, longitudinal, objective data have validated physical activity findings, and actigraphy data are well suited to address this task. To evaluate whether differences in physical movement exist among individuals treated with SSRIs compared with control participants and to identify the unique features of the movement of patients treated with SSRIs. This cross-sectional study examines longitudinally collected wearable movement data within a cross-sectional sample of 7162 participants from the 2005-2006 National Health and Nutrition Examination Survey (NHANES), a nationally representative population-based sample of noninstitutionalized persons in the US having both medication information and passive movement data. Statistical analysis was performed from April 1, 2021, to February 1, 2022. The use of SSRIs (sertraline hydrochloride, escitalopram oxalate, fluoxetine hydrochloride, paroxetine hydrochloride, and citalopram hydrobromide), as reported by participants interviewed by NNHANES personnel, was the primary exposure, measured as a binary variable (taking an SSRI vs not taking an SSRI). The primary outcome was the intensity of body movement as recorded by a piezoelectric accelerometer worn on the right hip for more than 1 week. Of the 7162 participants included in the study, the mean (SD) age was 33.7 (22.6) years, 266 (3.7%) were taking an SSRI, 3706 (51.7%) were female, 1934 (27.0%) were Black, 1823 (25.5%) were Mexican American, 210 (2.9%) were other Hispanic, 336 (4.7%) were other or multiracial, and 2859 (39.9%) were White (per the NHANES data collection protocol). A cross-validated, deep learning classifier was constructed that achieved fair performance predicting SSRI use (area under the curve, 0.67 [95% CI, 0.64-0.71] for the validation set and 0.66 [95% CI, 0.64-0.68] for the test set). To account for possible confounding by indication, we constructed a parallel model incorporating depression severity, finding only marginal performance improvement. When averaged across individuals and across 7 days, the results show less overall movement in the SSRI group (mean, 120.1 vertical acceleration counts/min [95% CI, 115.7-124.6 vertical acceleration counts/min]) compared with the non-SSRI control group (mean, 168.8 vertical acceleration counts/min [95% CI, 162.8-174.9 vertical acceleration counts/min]). This cross-sectional study found a moderate association between passive movement and SSRI use, as well as SSRI detection capacity of passive movement using time series deep learning models. The results support the use of passive sensors for exploration and characterization of psychotropic medication adverse effects.