The practical utility of Møller-Plesset (MP) perturbation theory is severely constrained by the use of Hartree-Fock (HF) orbitals. It has recently been shown that the use of regularized orbital-optimized MP2 orbitals and scaling of MP3 energy could lead to a significant reduction in MP3 error [Bertels, L. W.; J. Phys. Chem. Lett. 2019, 10, 4170 4176]. In this work, we examine whether density functional theory (DFT)-optimized orbitals can be similarly employed to improve the performance of MP theory at both the MP2 and MP3 levels. We find that the use of DFT orbitals leads to significantly improved performance for prediction of thermochemistry, barrier heights, noncovalent interactions, and dipole moments relative to the standard HF-based MP theory. Indeed, MP3 (with or without scaling) with DFT orbitals is found to surpass the accuracy of coupled-cluster singles and doubles (CCSD) for several data sets. We also found that the results are not particularly functional sensitive in most cases (although range-separated hybrid functionals with low delocalization error perform the best). MP3 based on DFT orbitals thus appears to be an efficient, noniterative O(N6) scaling wave-function approach for single-reference electronic structure computations. Scaled MP2 with DFT orbitals is also found to be quite accurate in many cases, although modern double hybrid functionals are likely to be considerably more accurate.