As a candidate complementary to radio frequency (RF) communications, visible light communication (VLC) is a promising technology with the advantages of high date rate, low power consumption and high security. Since the small light emitting diode (LED) coverage may cause the frequent handover of users, cell-free network model with low handover overhead is utilized in this paper. Meanwhile, the massive and random deployment of users in cell-free VLC network may cause severe interference, which deteriorates the system performance. Then, for reducing interference, we put forward a joint AP Grouping and User Clustering for Interference Management (AGUC-IM) in cell-free VLC network. Firstly, the weighted undirected interference graph of AP is constructed according to user-AP access, and an AP grouping algorithm for group number optimization is designed for mitigating interference and improving resource utilization. Thereafter, a time–frequency resource optimization based on twice user clustering algorithm is designed to mitigate the residual interference of users and improve user satisfaction. Simulation results show that, compared with the existing algorithms, the proposed algorithm effectively improves the user satisfaction while significantly improving system throughput.