The combination of Additive Increase and Multiplicative Decrease (AIMD) congestion control and Random Early Detection (RED) queue as a whole congestion control system plays a key role in the overwhelming success of the Internet. Thus it is important to investigate the periodic oscillations and complicated dynamics of the state-dependent delayed homogeneous Additive Increase and Multiplicative Decrease/Random Early Detection (AIMD/RED) congestion control system and its modified version fully in this paper. Firstly employing the semi-analytical method called as the harmonic balance method with alternating frequency/time (HB-AFT) domain technique, the approximate analytical expressions of periodic solutions of the generalized homogeneous-flow Additive Increase and Multiplicative Decrease/Random Early Detection (AIMD/RED) system with state-dependent round-trip delay are considered. We compare them with the results of numerical simulations by WinPP, they agree very well with each other. It demonstrates that the method employed here is versatile, valid, simple and effective. Then to the end of improving its modeling and performance, we modify the above model by taking an easy approximate dropping function. Furthermore, for the modified delayed homogeneous system, the approximate analytical expressions of periodic solutions are obtained accurately, and some complex dynamics are also presented. Four kinds of bi-stability, i.e., the coexistence of chaos and Period-3 solution, that of Period-1 and Period-2 solutions, that of Period-2 and Period-2 solutions, that of Period-4 and Period-2 solutions are disclosed. And a route to chaos, i.e., Period Doubling bifurcation to chaos, and the window of Period-3 to chaos are also discovered. The periodic oscillation can reduce the link utilization, induce the TCP stream synchronization services and further congestion. Chaotic oscillation may result in collapse. Therefore, all complex dynamical phenomena found in this paper are harmful and should be avoided. The obtained results can be very helpful for the researchers to have a better understanding of the mechanism of the network congestion control system, and they can select the parameters properly to improve the network stability and performance.