ABSTRACT During vehicle operation, the brake system is influenced by wheel–rail interactions. To investigate the dynamic behaviours of the brake system, we propose a novel dynamic model of the brake system by considering the coupling effects between the disc–pad and the wheel–rail subsystems. Using the established model, the nonlinear coupled dynamic behaviours including wheel–rail adhesion characteristics, wheel–disc torsional vibrations and disc–pad stick–slip vibrations were comprehensively explored. The results showed that braking conditions remarkably influenced the longitudinal slip rate. The wheelset and brake units affect each other, and the wheels on one wheelset influence each other when they are asynchronous. The brake system is stable in the wheel–rail stick state, but becomes unstable in the wheel–rail stick–slip state or slip state. The limit cycle analysis showed that the brake pad stick–slip vibrations were easier to occur at a large slip rate.