According to the World Health Organization (WHO), about 1% of the world’s population are wheelchair users. Existing studies indicate that wheelchair users wait longer than others for the elevator. However, taking the elevator is often the only option for wheelchair users to travel across floors. This work proposes an Elevator Priority Scheduling (EPS) system taking into consideration of wheelchair users. With the recent progress of computer vision, existing surveillance cameras in buildings are convenient for occupants’ detection in public spaces. This work utilized the DeepSORT model to gather the information of elevator passenger waiting time, passenger number, and passenger type. Passenger priority weights were associated to the level of passenger needs: passengers with mobility impairment have higher priority weights than other passengers. An elevator simulator was implemented for the study of various passenger patterns. Results indicated that the proposed EPS out performed traditional elevator systems in terms of average waiting time for people with special needs. Sensitivity analysis also provided guide to parameter selection for the proposed EPS.