A circular approach to managing resources that will promote their reuse and recycling is nowadays of crucial importance for a sustainable society. In this regard, the substitution of natural aggregates by steel slag in construction materials represents a promising option. In this paper, the use of Ladle Furnace Slag (LFS) as sustainable filler and fine aggregate for asphalt mixtures is studied. In particular, the evaluation of the LFS contribution in mastic and mortar mixes at mid-range and high-service temperatures is investigated, employing a dynamic shear rheometer to assess the main viscoelastic properties as well as the fatigue and the permanent deformation resistance of the blends. The experimental findings showed that the addition of LFS led to a clear stiffening effect, altering the chemo-physical interaction with the bitumen and producing an appreciable difference in complex stiffness moduli and phase angles. Regardless of the aging condition, the use of LFS lowered the linear viscoelastic limits and increased the elasticity of blends in the case of both mastics and mortars. It caused also a slightly higher thermal dependence of the linear viscoelastic properties even if the enhanced stiffness and elasticity produced appreciable improvements in the permanent deformation resistance. In contrast, a slight reduction of fatigue resistance was observed under the test conditions and was reasonably ascribed to the higher stiffness of LFS blends. Further research is needed to strengthen these promising results and to address the issues at a multiscale level, in particular to evaluate possible lower workability and reduced ductility due to the encountered higher stiffness of slag-based materials.