We present a way to study a wide class of optimal design problems with a perimeter penalization. More precisely, we address existence and regularity properties of saddle points of energies of the form $$ (u,A) \quad \mapsto \quad \int_\Omega 2fu \; \text{d}x \; - \int_{\Omega \cap A} \sigma_1 \mathscr A u \cdot \mathscr A u \; \text{d}x \; - \int_{\Omega \setminus A} \sigma_2\mathscr A u\cdot \mathscr A u \; \text{d}x \; + \; \text{Per}(A;\overline \Omega),$$ where $\Omega$ is a bounded Lipschitz domain, $A\subset \mathbb R^N$ is a Borel set, $u:\Omega \subset \mathbb R^N \to \mathbb R^d$, $\mathscr A$ is an operator of gradient form, and $\sigma_1, \sigma_2$ are two not necessarily well-ordered symmetric tensors. The class of operators of gradient form includes scalar- and vector-valued gradients, symmetrized gradients, and higher order gradients. Therefore, our results may be applied to a wide range of problems in elasticity, conductivity or plasticity models. In this context and under mild assumptions on $f$, we show for a solution $(w,A)$, that the topological boundary of $A \cap \Omega$ is locally a $\rm{C}^1$-hypersurface up to a closed set of zero $\mathscr H^{N-1}$-measure.