In this paper, we combine bifurcation techniques and numerical simulations to fully analyze the nonlinear propagation of dust ion-acoustic waves in a complex dusty plasma consisting of warm adiabatic ions, nonextensive electrons, and negatively (positively) charged dust particles. An autonomous dynamical system is derived which captures the main features of traveling wave solutions and transition between nonlinear modes. It is shown that for a range of electron nonextensivity and negative (positive) dust concentration, there are different existence domains of stability regions. At the critical points of nonextensivity parameter and dust concentration, we show that the motion dynamics of low-frequency dust ion-acoustic traveling waves undergoes a transcritical bifurcation, where two equilibrium points coalesce, and then switch their stability. A new kind of solitary structure is also observed which is characterized by the transcritical bifurcation parameters with half-stable equilibrium points. It is found that an increase in negative (positive) dust concentration leads to the appearance of bifurcation points at larger (smaller) values of electron nonextensivity. In accordance with the phase portraits analysis, the coexistence of homoclinic orbits, nonlinear periodic and supernonlinear periodic orbits are also investigated for different conditions. It is shown that the nonextensive parameter and dust polarity play a crucial role in the transition between nonlinear traveling waves.