ABSTRACT Teeter bed separator (TBS) plays an important role in coal and mineral fines beneficiation. It is believed that a favorable dense concentrated bed is the key to ensuring the effective particle separation. This paper mainly presents the investigation of the flow characteristics of the dense bed and its enhancing effect on particle fines separation by CFD-DEM to reveal the complex separation behavior of TBS. First, the enhancing effect of the dense bed for separating real fine coal is experimentally studied. The specific simulation for particle motion is further verified by the literature experiment. The simulated dynamic behavior of the mismatch-prone particles reveals that the dense bed in the concentration range of 20–47% could promote particle separation by density. The fact is that a certain velocity difference is generated between mismatch-prone particles, and the smaller heavier particles move faster than the larger lighter particles. It is especially beneficial for separating fine coal material in TBS. The enhancing mechanism of the dense bed is further revealed by analyzing the mismatch-prone particle forces of collision and fluid drag force. Comprehensively, the appropriate bed concentration of 35–45% is suggested for the actual TBS operation.