This paper is concerned with an inverse transmission problem for recovering the shape of a penetrable rectangular grating sitting on a perfectly conducting plate. We consider a general transmission problem with the coefficient λ ≠ 1 which covers the transverse magnetic (TM) polarization case. It is proved that a rectangular grating profile can be uniquely determined by the near-field observation data incited by a single plane wave and measured on a line segment above the grating. In comparison with the transverse electric (TE) case (λ = 1), the wave field cannot lie in H 2 around each corner point, bringing essential difficulties in proving uniqueness with one plane wave. Our approach relies on singularity analysis for Helmholtz transmission problems in a right-corner domain and also provides an alternative idea for treating the TE transmission conditions which were considered in the authors’ previous work (Xiang and Hu 2023 Inverse Problems 39 055004).