Quantum computing is a promising paradigm that can provide viable solutions to high-complexity problems. The $k$-medoids algorithm is a powerful clustering method ubiquitously used in data mining, image processing, pattern recognition, etc. The core of $k$-medoids algorithm is to perform cluster assignment and center update, which are time-consuming for large data sets. A\"{\i}meur et al. proposed a quantum $k$-medoids algorithm [A\"{\i}meur, Brassard, and Gambs, Mach. Learn. 90, 261 (2013)] by quantizing the center update. Nevertheless, it has a query complexity $O({N}^{3/2})$ for one iteration, which is computationally expensive for a large $N$ where $N$ is the number of points. In this paper, we propose a complete quantum algorithm for $k$-medoids algorithm. Specifically, in cluster assignment, we devise a quantum subroutine to calculate the Manhattan distance between any two points and then assign all points to the closest center in parallel, which is faster than what is achievable classically. In center update, for a cluster, we use parallel amplitude estimation to calculate the average distance of each point to all the others. It makes our algorithm polynomially faster than the algorithm of A\"{\i}meur et al., whose sum of distances of each point to all the others is computed by adding the distances one by one. Our quantum $k$-medoids algorithm, with time complexity $\stackrel{\ifmmode \tilde{}\else \~{}\fi{}}{O}({N}^{1/2})$, achieves a polynomial speedup in $N$ compared to the existing one.