Travel behavior understanding is a long-standing and critically important topic in the area of smart cities. Big volumes of various GPS-based travel data can be easily collected, among which the taxi GPS trajectory data is a typical example. However, in GPS trajectory data, there is usually little information on travelers’ activities, thereby they can only support limited applications. Quite a few studies have been focused on enriching the semantic meaning for raw data, such as travel mode/purpose inferring. Unfortunately, trip purpose imputation receives relatively less attention and requires no real-time response. To narrow the gap, we propose a probabilistic two-phase framework named TripImputor , for making the real-time taxi trip purpose imputation and recommending services to passengers at their dropoff points. Specifically, in the first phase, we propose a two-stage clustering algorithm to identify candidate activity areas (CAAs) in the urban space. Then, we extract fine-granularity spatial and temporal patterns of human behaviors inside the CAAs from foursquare check-in data to approximate the priori probability for each activity, and compute the posterior probabilities (i.e., infer the trip purposes) using Bayes’ theorem. In the second phase, we take a sophisticated procedure that clusters historical dropoff points and matches the dropoff clusters and CAAs to immerse the real-time response. Finally, we evaluate the effectiveness and efficiency of the proposed two-phase framework using real-world data sets, which consist of road network, check-in data generated by over 38 000 users in one year, and the large-scale taxi trip data generated by over 19 000 taxis in a month in Manhattan, New York City, USA. Experimental results demonstrate that the system is able to infer the trip purpose accurately, and can provide recommendation results to passengers within 1.6 s in Manhattan on average, just using a single normal PC.