Kitaev’s quantum double model is a lattice realization of Dijkgraaf–Witten topological quantum field theory. Its topologically protected ground state space has broad applications for topological quantum computation and topological quantum memory. We investigate the Z2 symmetry enriched generalization of the model for the Abelian group in a categorical framework and present an explicit Hamiltonian construction. This model provides a lattice realization of the Z2 symmetry of the topological phase. We discuss in detail the categorical symmetry of the phase, for which the electric-magnetic (EM) duality symmetry is a special case. The aspects of symmetry defects are investigated using the G-crossed unitary braided fusion category. By determining the corresponding anyon condensation, the gapped boundaries and boundary-bulk duality are also investigated. In the last part, an explicit lattice realization of EM duality is discussed.