Speckle covariance matrix estimation plays an important role in adaptive target detection of maritime radars. Spatial heterogeneity of sea clutter incurs insufficient secondary data in estimation, which degrades detection performance. It is an effective way of estimation improvement to exploit the Doppler spectrum knowledge of sea clutter for structural dimension reduction of the speckle covariance matrix. This paper proposes a speckle covariance matrix estimation method based on the spectral symmetry knowledge of sea clutter, indicating that the spectrum of sea clutter is symmetric with respect to the Doppler offset. First of all, the spectral symmetry of sea clutter is analyzed by using the measured data from several opened databases and the results show that most of the data excluding sea spikes have symmetric Doppler spectra. Secondly, the spectral symmetry constrains the speckle covariance matrix into the Hadamard product of a special rank-one Toeplitz matrix and a real positive definite symmetric matrix. The degrees of freedom are reduced to almost half of that of a Hermitian matrix and thus fewer secondary data are required. Thirdly, a robust iterative maximum likelihood estimator is proposed to estimate the speckle covariance matrix with structural constraint and is free of the clutter texture distribution. The performance of the estimator is verified by the measured data and the results show that it is superior to traditional estimators, particularly in fewer secondary data.