In full-duplex orthogonal frequency-division multiplexing systems with IQ imbalances, a frequency-domain least-squares (FD-LS) channel estimator is proposed to estimate both source-to-destination (intended) and destination-to-destination (self-interference) channels. Subsequently, an optimal closed-form pilot matrix is derived to minimize the sum of mean square errors (sum-MSE) of the proposed FD-LS channel estimator. Then, an improved FD-LS estimator is presented and proved to further improve the performance of the FD-LS by exploiting the time-domain property of the channel. In the presence of channel estimation error, an optimal low-complexity maximum likelihood (ML) detector is developed by using eigenvalue and singular value decompositions and whitening the residual self-interference plus noise. Simulation results show that given the proposed FD-LS estimator, the optimal pilot matrix performs much better than those nonsingular pilot matrices with larger conditional numbers (CN). To be specific, the former achieves about 10-dB signal-to-noise ratio (SNR) gain over the latter with CN = 10, the improved FD-LS channel estimator provides about 6-dB SNR gain over the FD-LS estimator at a fixed bit error rate (BER), and the proposed whitening-filter ML detector performs at least 0.6 dB better than the conventional ML detector at a given BER of $10^{-3}$ in the medium-and high-SNR regions.