Mobile manipulation is essential for robots to accomplish everyday household chores such as set the table. In order to successfully perform manipulation tasks, a 3-D representation of the environment and the pose of the target object are needed. In this article, we propose a systematic solution for safe and efficient robot manipulation. Innovatively, a task-oriented environment modeling strategy is presented for collision-free navigation and motion planning, which integrates the grid-based 2-D map and local real-time octree-based 3-D representation. Furthermore, we introduce a task-oriented object pose estimation approach based on the fiducial marker and ontology technology. In particular, a property-driven target object inference and pose estimation algorithm is designed, which allows the mobile robot to implement the task-oriented object manipulation. The proposed solution is evaluated through real-world experiments, where memory usage, computation time, collision checking, and object pose estimation are extensively investigated. Also, application scenarios are presented to validate the effectiveness and efficiency of our proposal.