In recent years, hybrid design strategies combining machine learning (ML) with electromagnetic optimization algorithms have emerged as a new paradigm for the inverse design of photonic structures and devices. While a trained, data-driven neural network can rapidly identify solutions near the global optimum with a given data set’s design space, an iterative optimization algorithm can further refine the solution and overcome data set limitations. Furthermore, such hybrid ML-optimization methodologies can reduce computational costs and expedite the discovery of novel electromagnetic components. However, existing hybrid ML-optimization methods have yet to optimize across both materials and geometries in a single integrated and user-friendly environment. In addition, due to the challenge of acquiring large data sets for ML, as well as the exponential growth of isolated models being trained for photonics design, there is a need to standardize the ML-optimization workflow while making the pretrained models easily accessible. Motivated by these challenges, here we introduce DeepAdjoint, a general-purpose, open-source, and multiobjective “all-in-one” global photonics inverse design application framework that integrates pretrained deep generative networks with state-of-the-art electromagnetic optimization algorithms such as the adjoint variables method. DeepAdjoint allows a designer to specify an arbitrary optical design target, then obtain a photonic structure that is robust to fabrication tolerances and possesses the desired optical properties, all within a single user-guided application interface. We demonstrate DeepAdjoint for the design of infrared-controlled metasurfaces and show that a wide range of structures and absorption spectra can be achieved and optimized, including single- and multiresonance behavior through single- and supercell-class structures, respectively. Our framework, thus, paves a path toward the systematic unification of ML and optimization algorithms for photonic inverse design.