In recent years, personal mobility vehicles have been required to operate autonomously in places with numerous pedestrians. A navigation method using a single human-following scheme is used to avoid collision with pedestrians. However, in many cases, a single human-following method cannot be successfully used for guidance. In crowded places, pedestrians do not always keep walking in the desired direction a user wants to go, and the vehicle must change the target pedestrian frequently. Instead of following a single pedestrian, we propose a method for the vehicle to follow a cluster of pedestrians for stable and robust following. First, the pedestrians around the vehicle are detected by multiple RGB-D cameras, and the pedestrians are tracked using YOLO and Deep Sort. Pedestrians are classified according to their walking direction, and the cluster of pedestrians walking toward the goal is selected and followed. However, the position of pedestrian is sometimes lost in occlusions and the accuracy of the walking direction depends on the distance and pose detected by the sensors. A notable problem is that the cluster of pedestrians is unstable in the cluster following; therefore, a median of candidate vectors (MCV) observer is used to remove outliers caused by observation errors. The proposed method is applied to a scenario involving pedestrians walking toward an elevator hall in a building, and its effectiveness is verified through experiments.