Let $\mu_2(\Omega)$ be the first positive eigenvalue of the Neumann Laplacian in a bounded domain $\Omega\subset\mathbb{R}^N$. It was proved by Szeg\H{o} for $N=2$ and by Weinberger for $N \geq 2$ that among all equimeasurable domains $\mu_2(\Omega)$ attains its global maximum if $\Omega$ is a ball. In the present work, we develop the approach of Weinberger in two directions. Firstly, we refine the Szeg\H{o}-Weinberger result for a class of domains of the form $\Omega_{\text{out}}\setminus\overline{\Omega}_{\text{in}}$ which are either centrally symmetric or symmetric of order $2$ (with respect to every coordinate plane $(x_i,x_j)$) by showing that $\mu_{2}(\Omega_{\text{out}}\setminus\overline{\Omega}_{\text{in}})\leq\mu_2(B_\beta\setminus\overline{B}_\alpha)$, where $B_\alpha, B_\beta$ are balls centered at the origin such that $B_\alpha\subset\Omega_{\text{in}}$ and $|\Omega_{\text{out}}\setminus\overline{\Omega}_{\text{in}}|=|B_\beta\setminus\overline{B}_\alpha|$. Secondly, we provide Szeg\H{o}-Weinberger type inequalities for higher eigenvalues by imposing additional symmetry assumptions on the domain. Namely, if $\Omega_{\text{out}}\setminus\overline{\Omega}_{\text{in}}$ is symmetric of order $4$, then we prove $\mu_{i}(\Omega_{\text{out}}\setminus\overline{\Omega}_{\text{in}})\leq\mu_i(B_\beta\setminus\overline{B}_\alpha)$ for $i=3,\dots,N+2$, where we also allow $\Omega_{\text{in}}$ and $B_\alpha$ to be empty. If $N=2$ and the domain is symmetric of order $8$, then the latter inequality persists for $i=5$. Counterexamples to the obtained inequalities for domains outside of the considered symmetry classes are given. The existence and properties of nonradial domains with required symmetries in higher dimensions are discussed. As an auxiliary result, we obtain the non-radiality of the eigenfunctions associated to $\mu_{N+2}(B_\beta\setminus\overline{B}_\alpha)$.