The consensus problem of heterogeneous linear multi-agent systems (MASs) with communication, input and output time-delays is studied in this paper. An observer-predictor is designed to estimate the future states of each agent, and then a distributed dynamic controller based on this observer-predictor is proposed to solve the concerned problem. Switching and directed communication topologies are considered in this work. It is shown that the closed loop control systems can reach consensus for arbitrarily large yet bounded time-varying communication delays, time-varying input delays and constant output delays. A simulation example is provided to illustrate the effectiveness of the controller.