In the Discrete Element modelling of soils, the micromechanical behaviour at contacts has often been considered to have a minor influence on the macromechanical response, leaving basic theoretical models (e.g., Hertz) to describe the contact response in the normal direction. A realistic description of the contact response may be crucial especially when investigating small strain ranges. Recent experimental inter-particle loading tests on gravel suggest that the classic contact models fail to reproduce certain mechanical features, especially when roughness is significant. Here, some of these experimental observations, including a softer response than the Hertz model under loading in the normal direction, and plasticity on unloading, are implemented in a DEM model for the simulation of small strain tests on railway ballast. The influence of these features on small strain stiffness is highlighted. A micromechanical analysis is carried out to show how each of the contact-level features introduced affects the macroscopic response. • Classic contact models do not work well for rough surfaces. • Accounting for lower normal stiffness due to contact with asperities and plasticity. • True contact behaviour implemented in numerical DEM simulations of ballast. • Contact behaviour can explain small strain stiffness degradation.