A novel ultracompact two-dimensional (2D) waveguide-based metasurface is proposed herein and applied for the first time to reduce mutual coupling in antenna array for multiple-input multiple-output applications. The unit cell of the proposed 2D waveguide-based metasurface is ultracompact (8.6 mm × 4.8 mm, equal to λ0/14.2 × λ0/25.5) mainly due to the symmetrical spiral lines etched on the ground. The metasurface exhibits a bandgap with two transmission zeros attributing to the negative permeability in the vicinity of magnetic resonance and the negative permittivity in the vicinity of electric resonance. Taking advantage of these two features, a microstrip antenna array is then designed, fabricated, and measured by embedding an 8 × 1 array of the well-engineered 2D waveguide-based metasurface elements between two closely spaced (9.2 mm, equal to λ0/13.3) H-plane coupled rectangular patches. There is good agreement between the simulated and measured results, indicating that the metasurface effectively reduces antenna mutual coupling by more than 11.18 dB and improves forward gain. The proposed compact structure has one of the highest reported decoupling efficiencies among similar periodic structures with comparable dimensions. © 2015 Wiley Periodicals, Inc. Int J RF and Microwave CAE 25:789–794, 2015.