The structure of the doubly magic $^{132}_{50}$Sn$_{82}$ has been investigated at the ISOLDE facility at CERN, populated both by the $\beta^-$decay of $^{132}$In and $\beta^-$-delayed neutron emission of $^{133}$In. The level scheme of $^{132}$Sn is greatly expanded with the addition of 68 $\gamma$-transitions and 17 levels observed for the first time in the $\beta$ decay. The information on the excited structure is completed by new $\gamma$-transitions and states populated in the $\beta$-n decay of $^{133}$In. Improved delayed neutron emission probabilities are obtained both for $^{132}$In and $^{133}$In. Level lifetimes are measured via the Advanced Time-Delayed $\beta\gamma\gamma$(t) fast-timing method. An interpretation of the level structure is given based on the experimental findings and the particle-hole configurations arising from core excitations both from the \textit{N} = 82 and \textit{Z} = 50 shells, leading to positive and negative parity particle-hole multiplets. The experimental information provides new data to challenge the theoretical description of $^{132}$Sn.