Introduction: Existing theories and empirical works link phone use with anxiety; however, most leverage subjective self-reports of phone use (e.g., validated questionnaires) that may not correspond well with true behavior. Moreover, most works linking phone use with anxiety do not interrogate associations within a temporal framework. Accordingly, the present study sought to investigate the utility of passively sensed phone use as a longitudinal predictor of anxiety symptomatology within a population particularly vulnerable to experiencing anxiety. Methods: Using data from the GLOBEM study, which continuously collected longitudinal behavioral data from a college cohort of N = 330 students, weekly PHQ-4 anxiety subscale scores across 3 years (2019–2021) were paired with median daily phone use records from the 2 weeks prior to anxiety self-report completion. Phone use was operationalized through unlock duration which was passively curated via Apple’s “Screen Time” feature. GPS-tracked location data was further utilized to specify whether an individual’s phone use was at home or away from home. Within-individual and temporal associations between phone use and anxiety were modeled within an ordinal mixed-effects logistic regression framework. Results: While there was no significant association between anxiety levels and either median total phone use or median phone use at home, participants in the top quartile of median phone use away from home were predicted to exhibit clinically significant anxiety levels 20% more frequently than participants in the bottom quartile during the first study year; however, this association weakened across successive years. Importantly, these associations remained after controlling for age, physical activity, sleep, and baseline anxiety levels and were not recapitulated when operationalizing phone use with unlock frequency. Conclusions: These findings suggest that phone use may be leveraged as a means of mitigating or coping with anxiety in social situations outside the home, while pandemic-related developments may also have attenuated this behavior later in the study. Nevertheless, the present results suggest promise in interrogating a larger suite of objectively measured phone use behaviors within the context of social anxiety.