This study studies the exponential state consensus problem for hierarchical multi-agent dynamical systems with switching topology and inter-layer communication delay. A Lyapunov function is introduced to quantify the total disagreement among the nodes of a hierarchical multi-agent system. By using the stability theory of switched system and graph theory of hierarchical network topology, delay dependent sufficient conditions are proposed to achieve the exponential hierarchical average consensus with the average dwell time of the switching law larger than a minimum bound. Illustrative examples are provided to verify the effectiveness of the proposed approach.