Granular media generally exhibit fluid phase, solid phase, or two coexisting phases, and the mechanism of the phase transition is not clear enough. Therefore, unlike the airflow-structure or fluid–structure coupled vibrations, the granules-structure coupled vibrations have not yet been theoretically modeled, and the interest in such coupled vibrations, which has just started in recent years, is mainly focused on experiments and simulations. In order to develop a theoretical model of the coupled vibrations of granular media and beam structures, in this paper, a novel concept called the inertia-viscous effect is introduced in terms of terminology and mathematical expression, and surprisingly, its constitutive relation can be summarized in a unified framework based on fractional calculus, together with Hooke spring, Newtonian dashpot, and Abel sticky pot. Furthermore, the existence of this effect in the granules-beam coupled vibrations is experimentally confirmed, and more importantly, the effect is more suitable for modeling by fractional-order derivatives rather than the conventional integer-order ones due to the frequency-dependent characteristics of the effect. Based on this, the theoretical model of the granules-beam coupled vibrations and the associated parameters identification procedure are successfully developed, in which a fractional-order model is proposed to describe the additional dynamic load (ADL) of the granular media on the beam. The numerical and experimental results suggest that the system response can be accurately predicted by the developed theoretical model without the computational cost as in the discrete element methods, which is the main contribution of this work. Finally, the effectiveness and robustness of the developed theoretical model are further verified by the results of numerous parametric experiments, and importantly, by combining these results with the analytical results, the mechanism of ADL is revealed, thus providing insights for modeling coupled vibrations between other engineering structures and granular media. From another perspective, this work can also be seen as a novel application of fractional calculus in the dynamics of mechanical systems.