The promotion of physical activity (PA) in various subgroups of the population such as people with physical disabilities has been spotlighted in the revised guidelines of The World Health Organization (WHO), Geneva, Switzerland. In order to update public health interventions, there is a need to identify factors that may promote or prevent engagement in PA for special subgroups of the population. This study aims to calculate the PA levels of individuals with and without physical disabilities in Saudi Arabia, their assessment of the environmental (EQoL), and the predictive role of EQoL in PA. The International Physical Activity Questionnaire (IPAQ) and the EQoL domain in standardized WHOQoL Questionnaire were administered on both groups of the population. The study sample comprised 116 individuals with physical disabilities and 243 individuals without any form of disability as a control group. A regression analysis was performed to analyze the predictors of PA in both groups. Findings showed that among the individuals with disabilities, older people were more likely to engage in PA as compared to the younger age group (p < 0.05) and males were significantly less likely to meet the PA criteria. Some of the EQoL features such as safety increased the likelihood of PA up to 2.3 times (p < 0.05) in individuals with physical disabilities. In addition, opportunities for leisure activities were a significant predictor of PA among both groups of individuals with and without physical disabilities (p < 0.05). Our findings suggest that upcoming public health interventions should focus on improving various dimensions of EQoL for the promotion of physical activity among individuals with physical disabilities. Additional studies are needed to further explore various sociodemographic and environmental factors which can affect the PA status of disabled groups.