The stability of a braking system is crucial for the steady and safe operation of trains. To investigate the dynamic behaviors and stability of a train braking system as influenced by key parameters, a numerical model integrating mode-coupling, negative friction–velocity slope (NFVS), and stick–slip theories was established. Additionally, the wheelset connected with the braking system was contained in the model considering that mode-coupling and stick–slip characteristics were affected by system structures. Based on this model, the influence and mechanism of stiffness, damping, and NFVS characteristics on mode-coupling of the braking system were revealed. The results showed that when the tangential and normal contact stiffness values of the pad were close, the system experienced mode-coupling instability and vibrations in these two directions were coupled. When the tangential and normal damping values of the pad were unequal, the energy dissipation in the two directions became different and the system instability was enhanced. Additionally, the system complex eigenvalues always had positive real parts and the dynamic behaviors were more complicated by considering NFVS at relatively low speeds. This is because NFVS characteristics can be equivalent to negative damping values, which cause the difference in tangential and normal damping values and continuously input energy into the braking system, then weakening the system stability.