In direction-of-arrival (DOA) estimation with sensor arrays, the background noise is usually modeled to be uncorrelated uniform white noise, such that the related algorithms can be greatly simplified by making use of the property of the noise covariance matrix being a diagonal matrix with identical diagonal entries. However, this model can be easily violated by the nonuniformity of sensor noise and the presence of outliers that may arise from unexpected impulsive noise. To tackle this problem, we first introduce an exploratory factor analysis (EFA) model for DOA estimation in nonuniform noise. Then, to deal with the outliers, a generalized extreme Studentized deviate (ESD) test is applied for outlier detection and trimming. Based on the trimmed data matrix, a modified EFA model, which belongs to weighted least-squares (WLS) fitting problems, is presented. Furthermore, a monotonic convergent iterative reweighted least-squares (IRLS) algorithm, called the iterative majorization approach, is introduced to solve the WLS problem. Simulation results show that the proposed algorithm offers improved robustness against nonuniform noise and observation outliers over traditional algorithms.