The energy consumption management of small base stations (SBSs) in wireless caching networks with dense deployment of SBSs is an urgent problem to be solved. This paper jointly optimizes the SBS state control and user association problems in caching networks to reduce network energy consumption while taking into account the average service latency of the network to ensure user experience. First, a new definition of three-state SBSs in caching networks is proposed based on their ability to keep content cache updated. Then, a relaxed threshold setting method is designed and the SBS traffic prediction is used to obtain the initial state information of SBSs in the next period. In order to eliminate the impact on the accessing users when the switched-off base station (BS) wakes up, a SBS state asynchronous switching mechanism is proposed to ensure that the users who switch to the waking SBS can carry out communication services normally, and a user association strategy is constructed with the SBS load as the optimization target. Finally, a joint optimization model of user association and SBS state control (SSC-UA) is constructed to admit and correct the initial state of SBSs to maximize the system gain and obtain the final state strategy for each SBS in the next period. The simulation results show that the proposed SSC-UA algorithm can effectively improve the energy efficiency and reduce the network service delay at the same time.