Optical metasurfaces have been recently explored as ultrathin analog image differentiators. By tailoring the momentum transfer function, they can perform efficient Fourier filtering—and thus potentially any linear mathematical operation—on an input image, replacing bulky 4f systems. While this approach has been investigated in different platforms, and a few techniques have been explored to achieve the required angular response, little effort has been devoted so far to tailor and control also the polarization response of an image-processing metasurface. Here, we show that edge-detection metasurfaces can be designed with tailored polarization responses, while simultaneously preserving an isotropic response. In particular, we demonstrate numerically and experimentally single-layer silicon metasurfaces yielding efficient Laplacian operation on a 2D image with either large polarization asymmetry, or nearly polarization-independent response. In the former case, we show that a strongly asymmetric polarization response can be used to unlock more sophisticated on-the-fly image processing functionalities, such as dynamically tunable direction-dependent edge detection. In parallel, metasurfaces with dual-polarized response are shown to enable efficient operation for unpolarized or arbitrarily polarized images, ensuring high efficiency. For both devices, we demonstrate edge detection within relatively large numerical apertures and with excellent isotropy and intensity throughput. Our study paves the way for the broad use of optical metasurfaces for sophisticated, massively parallel analog image processing with zero energy requirements.