A calibration method to compensate for the mutual coupling and channel imbalance in a compact multi-input multi-output (MIMO) radar array based on turn table measurements is proposed. Two mutual coupling matrices, one for the transmit array and the other for the receive array, are used to model the relationship between the ideal manifold and the measured manifold. The practical factors such as the nonisotropic element pattern and the displacement of the array phase center are also taken into account. All the unknowns are determined by solving the nonlinear least-squares fitting problem iteratively. Using two coupling matrices consists of the mutual coupling mechanism in an MIMO array and has better calibration performance than using one matrix for the virtual array. It also reduces the dimension of the matrix and hence the computation load of calibration. Measurements by a microstrip array with two transmitters and four receivers are used to verify its effectiveness in angle estimation, beamforming, and superresolution of close targets.