Ellipsometry is a standard metrology tool for measuring thin films and characterizing surfaces. As an extension of single-point ellipsometry, imaging ellipsometry is implemented by replacing a detector with an imaging system and a detection array. The compensator used in an ellipsometer is often nonideal. In single point ellipsometry, the beam is aligned in an effort to pass it through the center of the compensator, and the retardation error is considered to remain near-constant during compensator rotation. In imaging ellipsometry, however, the retardation of the imperfect compensator manifests inhomogeneity across the whole view field, and then, the rotation of the compensator changes the error distribution for the same view field. In this paper, we describe imaging ellipsometry with polarizer and imperfect compensator rotations and propose a method of calibrating the inhomogeneous retardation of the compensator. We simultaneously compensate the alignment errors of the compensator and polarizer. The method can extract calibration parameters directly from one measurement, without the need for a separate calibration step. The approach was tested practically using a homemade imaging ellipsometer by generating maps of the ellipsometric parameters Δ and ψ of Si and SiO2/Si wafers.