In this work, we investigate the optimization of Hartree-Fock (HF) orbitals with our recently proposed combined first- and second-order (SO-SCI) method, which was originally developed for multi-configuration self-consistent field (MCSCF) and complete active space SCF (CASSCF) calculations. In MCSCF/CASSCF, it unites a second-order optimization of the active orbitals with a Fock-based first-order treatment of the remaining closed-virtual orbital rotations. In the case of the single-determinant wavefunctions, the active space is replaced by a preselected "second-order domain," and all rotations involving orbitals in this subspace are treated at second-order. The method has been implemented for spin-restricted and spin-unrestricted Hartree-Fock (RHF, UHF), configuration-averaged Hartree-Fock (CAHF), as well as Kohn-Sham (KS) density functional theory (RKS, UKS). For each of these cases, various choices of the second-order domain have been tested, and appropriate defaults are proposed. The performance of the method is demonstrated for several transition metal complexes. It is shown that the SO-SCI optimization provides faster and more robust convergence than the standard SCF procedure but requires, in many cases, even less computation time. In difficult cases, the SO-SCI method not only speeds up convergence but also avoids convergence to saddle-points. Furthermore, it helps to find spin-symmetry broken solutions in the cases of UHF or UKS. In the case of CAHF, convergence can also be significantly improved as compared to a previous SCF implementation. This is particularly important for multi-center cases with two or more equal heavy atoms. The performance is demonstrated for various two-center complexes with different lanthanide atoms.