Dual-stage, Q-learning based, Shortest-path Routing Computing Scheme: Evaluated A Case Study on Illegal-parking Surveillance Networks

Abstract

This paper explores a reinforcement learning (with Q-learning as the algorithm) for vehicle routing in a city-wide road network with a large customer (a generic name used for each of the locations (to denote a person possibly associated with this location) through which the vehicle needs to be routed) counts. Most reported research on vehicle routing had mainly focused on calculating the shortest route (while neglecting the shortest path between certain customer(s) and certain/all other customer(s)) between the customers. Hence, we proposed a dual-stage, Q-learning based, shortest-route generation (DQSR) in a road network, considering the shortest path between each customer and certain other customers. Notably, the first stage’s Q-learning agent develops the meta-graph of the shortest path between each customer and this customer’s nearest neighbor customers (chosen differently in number), instead of each other customer, and finds this path’s length. This development is motivated by the fact that any two adjacent customers in the shortest route are located closer than far apart, reducing the time complexity of the first stage’s Q-learning. Subsequently, the second stage’s Q-learning agent finds the shortest route from any customer to other customers from the meta-graph, connecting each other customer only once and returning to the starting customer. Hence, the DQSR generates the shortest route in a relatively shorter duration due to the small size of the state-action pairs in the meta-graph. Further, we conducted a case study on the DQSR-based, illegal-parking surveillance in Yeonsu-gu, Incheon, South Korea, to check the DQSR’s effectiveness. The case study demonstrates that the DQSR is more efficient than two other algorithms used for the same purpose in the case study, in terms of time complexity and shortest-route generation.