As a popular image manipulation technique, object removal can be achieved by image-inpainting without any noticeable traces, which poses huge challenges to passive image forensics. The existing detection approach utilizes full search for block matching, resulting in high computational complexity. This paper presents an efficient forgery detection algorithm for object removal by exemplar-based inpainting, which integrates central pixel mapping (CPM), greatest zero-connectivity component labeling (GZCL) and fragment splicing detection (FSD). CPM speeds up suspicious block search by efficiently matching those blocks with similar hash values and then finding the suspicious pairs. To improve the detection precision, GZCL is used to mark the tampered pixels in suspected block pairs. FSD is adopted to distinguish and locate tampered regions from its best-match regions. Experimental results show that the proposed algorithm can reduce up to 90% of the processing time and maintain a detection precision above 85% under different kinds of object-removed images.