Abstract Sum frequency generation (SFG) has multiple applications, from optical sources to imaging, where efficient conversion requires either long interaction distances or large field concentrations in a quadratic nonlinear material. Metasurfaces provide an essential avenue to enhanced SFG due to resonance with extreme field enhancements with an integrated ultrathin platform. In this work, we formulate a general theoretical framework for multi-objective topology optimization of nanopatterned metasurfaces that facilitate high-efficiency SFG and simultaneously select the emitted direction and tailor the metasurface polarization response. Based on this framework, we present novel metasurface designs showcasing ultimate flexibility in transforming the outgoing nonlinearly generated light for applications spanning from imaging to polarimetry. For example, one of our metasurfaces produces highly polarized and directional SFG emission with an efficiency of over 0.2 cm2 GW−1 in a 10 nm signal operating bandwidth.