In the UAV-based urban delivery setting, the battery capacity serves as one of the key factors affecting the delivery range of drones. For a typical multi-rotor unmanned aerial vehicle (UAV), the maximum effective working range of a lithium battery for a single flight leg is approximately 20 km. In practice, the lithium battery is replaced each time the UAV completes a delivery, even though the remaining power of the replaced lithium battery is still capable of supporting the following flight legs. Therefore, the one-order-one-swapping strategy significantly brings down the battery utilisation as well as the efficiency of the delivery system. In the present paper, the UAV-based delivery problem with battery swapping decision is considered, where multiple en route pickup and deliveries are allowed. To obtain optimal decisions in terms of flight route, time and location of each battery swapping, a branch-and-price-and-cut algorithm is introduced. First we formulate this problem as a mixed integer linear programming (MILP) model, and then decompose it into a master problem and a pricing sub-problem, where the master problem is considered as a set covering model, and the sub-problem as an elementary shortest path problem with resource constraints (ESPPRC). To speed up the process of column generation, efficient heuristics are also integrated to provide basic feasible solutions. In the pricing sub-problem, a combination of heuristic algorithm and dynamic programming is implemented to improve the computational efficiency. In the numerical experiments, the proposed algorithm is tested on various scales of UAV-based urban delivery instances, the performance of which is analysed and compared with the off-the-shelf commercial solver.