It is well known that gravitational waves distort equilibrium matter globally, making them amenable to detection with laser interferometers. Less well known is the fact that gravitational waves create local nonequilibrium stresses inside matter, which could conceivably lead to alternative detection methods. The gravitational wave-to-matter coupling $\ensuremath{\kappa}$ is a transport coefficient depending on the material and is poorly known for most substances. In the present work, we calculate $\ensuremath{\kappa}$ for a superfluid Fermi gas near unitarity using large-$N$ techniques, finding $\ensuremath{\kappa}=\frac{n}{12m}$, with $n$ the number density and $m$ the mass of the fermion, matching the result for free Dirac fermions at zero temperature. Our prediction is amenable to nonperturbative theoretical as well as experimental tests.