We study the entanglement entropy of gapped phases of matter in three spatial dimensions. We focus in particular on size-independent contributions to the entropy across entanglement surfaces of arbitrary topologies. We show that for low energy fixed-point theories, the constant part of the entanglement entropy across any surface can be reduced to a linear combination of the entropies across a sphere and a torus. We first derive our results using strong sub-additivity inequalities along with assumptions about the entanglement entropy of fixed-point models, and identify the topological contribution by considering the renormalization group flow; in this way we give an explicit definition of topological entanglement entropy $S_{\mathrm{topo}}$ in (3+1)D, which sharpens previous results. We illustrate our results using several concrete examples and independent calculations, and show adding "twist" terms to the Lagrangian can change $S_{\mathrm{topo}}$ in (3+1)D. For the generalized Walker-Wang models, we find that the ground state degeneracy on a 3-torus is given by $\exp(-3S_{\mathrm{topo}}[T^2])$ in terms of the topological entanglement entropy across a 2-torus. We conjecture that a similar relationship holds for Abelian theories in $(d+1)$ dimensional spacetime, with the ground state degeneracy on the $d$-torus given by $\exp(-dS_{\mathrm{topo}}[T^{d-1}])$.