The association between physical frailty and performance in different cognitive domains in the absence of cognitive disorders is poorly understood. Hence, we aimed to explore the associations between frailty levels based on the Fried Physical Frailty Phenotype and performance of different cognitive domains. We also aimed to examine the associations between cognitive function and each criterion in the Fried Frailty Scale using the same cognitive domains in a non-dementia population aged 60-96 years. This cross-sectional study included 4,329 participants aged 60-96 years, drawn from the "Good Aging in Skåne" population study. Frailty indices included handgrip strength, physical endurance, body mass index (BMI), physical activity, and walking speed. Cognitive function was assessed across eight domains: episodic memory, processing speed, semantic memory, verbal fluency, working memory, attention, executive function, and visual perception. We constructed adjusted multiple linear regression models for each cognitive domain, with the frailty levels as the independent variable. Likewise, we constructed linear regression models with each cognitive domain as the dependent variable and frailty criteria as independent variables. Physical frailty was associated with poor performance in episodic memory, processing speed, semantic memory, verbal fluency, working memory, attention, and executive functions (p<0.001 for all associations). Weaker hand grip strength was independently associated with poorer performance in all cognitive domains (p < 0.015). Higher levels of frailty were associated with poorer performance in all cognitive domains except visual perception. Describing frailty by considering cognitive functioning may provide a better understanding of frailty.