With the ever-growing demand for even higher throughput, ultradense networks (UDNs) are being deployed for the fifth generation (5G) mobile communications. Although massively distributed radio access points (APs) result in a considerable increase in throughput, they also cause some critical problems. When employing a wireless backhaul, the backhaul capacity becomes a limiting factor, which may result in a high packet loss rate. Furthermore, dense deployment of APs leads to more frequent handoffs for mobile user equipments, which results in heavy measurement and signaling overhead. To address the problem of frequent handoffs, virtual cell (VC) has been considered as a promising solution. However, the limited wireless backhaul capacity encountered by inflexible VC design may still result in an intolerable packet loss rate. For a better trade-off between the packet loss rate and the handoff overhead, a machine learning approach for flexible VC design is proposed that leverages particle swarm optimization (PSO) to quickly find the optimal VC solution. To be responsive to the dynamic traffic demand and backhaul capacity of APs, a new parameter called “weighted distance” is employed in the modified K-means algorithm, which is nested in the PSO procedures for master AP selection and VC boundary determination. Compared with an exhaustive search, optimal VC solutions can be found efficiently through considerably fewer iterations. The proposed method is generic and applicable to disparate UDN application scenarios.