This paper studies dynamic panel models with a factor error structure that is correlated with the regressors. Both short panels (small T) and long panels (large T) are considered. A dynamic panel forms a simultaneous-equation system, and under the factor error structure, there exist constraints between the mean and the covariance matrix. We explore the constraints through a quasi-FIML (full information maximum likelihood) approach. The quasi-FIML approach does not estimate individual effects, even if they are fixed constants, thus circumventing the incidental parameters problem in the cross-sectional dimension.The factor process is treated as parameters and it can have arbitrary dynamics. We show that there is no incidental parameters bias, for fixed or large T, and that the estimator is centered at zero even when scaled by the fast convergence rate of root-NT. We also study the efficiency of the quasi-FIML estimator. Finally, we develop a feasible and fast algorithm for computing the quasi-FIML estimators under interactive effects.