AbstractPrevious studies on topology optimization against exterior sound radiation have mainly focused on free‐face and half‐face problems. In contrast, the waveguide sound radiation problem is a finite domain problem, and the topology optimization work on the latter is challenging due to the complexity of multiple physical couplings. This paper presents a topology optimization approach for designing bi‐material distributions on vibrating shell structures in the shallow sea waveguide. The combined method of the finite element method, the virtual mass method, and the image method‐based boundary element method is adopted for the system response analysis, where the coupling of the direct sound waves and the reflected sound waves from the free sea surface and the rigid seabed, as well as the strong interaction between the structural and fluid domains, are considered. In this context, a new sensitivity analysis of waveguide sound pressure is proposed. In the optimization problem, the design variables are the volumetric densities of design material elements in a bi‐material model constructed by the solid isotropic material with penalization method, and the minimization of sound pressure level at a reference field point in the waveguide domain is taken as the design objective. Numerical examples are provided to illustrate the correctness of the sensitivity analysis approach and the validity of the proposed optimization procedure, and several factors on the optimized designs are also discussed.