In this paper, exponential propagator (EP) is introduced to the discontinuous Galerkin finite element method (DG-FEM) for solving transient electromagnetic problems. As a kind of explicit time-stepping method, the proposed EP has fourth order accuracy in time and requires only one storage unit for each of the degrees-of-freedom. It provides an attractive alternative to the high-order low-storage explicit Runge-Kutta (LSERK) approach, which is the most popular time-marching scheme currently used in high-fidelity simulation of Maxwell's equations with DG-FEM. When centered numerical flux is utilized, the discretized Maxwell's system in lossless region is symplectic structure-preserving (energy conserving), and thus very suitable for accurately computing the long-time propagation of waves in non-dissipative medium, or simulating the motion of charged particles in electromagnetic fields. Our method can also handle upwind numerical flux at the sacrifice of symplectic-conserving virtue for higher accuracy in spatial discretization and spurious inhibition. A major contribution of this work is that, the application range of this scheme is extended to lossy media such as conductors and particularly unsplit perfectly matched layers in the framework of DG-FEM. This greatly facilitates the treatment of open boundary conditions in scattering and/or radiation problems. Extensive numerical examples, including the analysis of photonic crystals, dielectric coated perfect conducting cylinder, propagation of waves in zero-index materials, and corrugated O-type Cherenkov oscillators etc., are presented to demonstrate its stability, accuracy, and performance. All the numerical experiments show that the presented method is superior to the LSERK scheme when centered numerical flux is adopted.