Occupant activity recognition (OAR) is essential for building management systems (BMS) to provide occupants with intelligent and comfort environments. Conventional sensing methodologies rely on burdensome wearables, privacy-risking cameras or specialized wireless devices. Pervasive existing Wi-Fi signals are a promising alternative and enable ubiquitous occupant sensing. In this paper, we propose a Wi-Fi-based OAR system called Wi-OAR that enables energy-efficient and user-centric services in smart offices. Its technical novelties are twofold. First, to recover activity-induced information, we innovatively present the fast and robust target component separation (FRTCS) algorithm regarding both time efficiency and high accuracy. Second, noting that handcrafted features can be inefficient and redundant, we develop an efficient feature selection algorithm based on class differences and information entropy. We prototyped the Wi-OAR system with only a pair of commercial Wi-Fi devices and implemented it in diverse office environments. The experimental results illustrate a consistent accuracy of over 96% in the different scenarios with considerable time cost savings. Further studies compare the system performance with prior approaches and discuss the influences of variables, which demonstrate the superiority of Wi-OAR.