In this paper, the harmonic balance method with the alternating frequency/time (HB-AFT) domain technique is extended to the dynamical systems with state-dependent delays and non-smooth right-hand side for the first time. Two types of network congestion control models [the modified transmission control protocol-random early detection (TCP-RED) model and the fluid-flow TCP-additive increase multiplicative decease (AIMD)/RED model] with state-dependent round-trip time delays and non-smooth right-hand side are considered in detail. First, their dynamics and bifurcation are analyzed by the numerical analysis method. Then, the analytical approximate expressions of the periodic solutions are obtained by employing the semi-analytical method named as HB-AFT. The results of the numerical simulation and HB-AFT agree with each other very well. It indicates that the HB-AFT technique is simple, valid, effective, and accurate for the non-smooth dynamical systems with state-dependent time delays. Besides, more complicated and rich dynamical phenomena, such as period-2 solution, period-3 solution, chaos and the period-doubling bifurcation, and the window of period-3 solution, which lead to chaos, are explored in the TCP-AIMD/RED model. Moreover, two different kinds of multi-stabilities, such as the coexistence of a stable period-1 solution, a stable period-3 solution, and an asymptotically stable equilibrium, and that of a stable period-1 solution, a stable period-2 solution, chaos, and an asymptotically stable equilibrium, are also discovered in the TCP-AIMD/RED congestion control model. It discloses that the interaction of the state-dependent delay and non-smooth function can easily induce the richer and more complicated dynamics. It can explain the complicated dynamical phenomena in the Internet network. Thus, the control and parameters tuning based on the above results could be used to avoid the congestion and optimize the performance. It can give the theoretical guidance to the application and research in network congestion control, which is very important in practical application.