In this paper, compact time-domain (TD) models featuring materials with frequency-dependent electromagnetic (EM) properties are derived. The considered frequency-dependent material models include multiterm Debye and Lorentz models for the electric permittivity and the magnetic permeability and a multiterm Drude model for the electric conductivity. The TD models are based on finite-element systems in the frequency domain (FD). To render the model compact and computationally efficient, the dimension of the FD system is compressed with the help of projection-based model-order reduction. In contrast to older approaches, the TD transformation is performed on the FD model itself rather than on the transfer function. The result is a state-space representation, which may either be solved by custom time integrators or imported into commercial circuit simulators. The advantages of the new approach include provable passivity of the FD model, provable causality of the TD model, and the ability to reconstruct the transient EM fields.