The precoding scheme based on codebooks is used to save the same set of codebook in advance at the transmitter and the receiver, then, the receiver selects the most appropriate precoding matrix from codebooks according to different channel state information (CSI). Therefore, the design of codebook plays an important role in the performance of the whole scheme. The symmetry-based hybrid precoder and combiner is a highly energy efficient structure in the millimeter-wave massive multiple-input multiple-output (MIMO) system, but at the same time, it also has the problems of high bit error rate and low spectral efficiency. In order to improve the spectral efficiency, we formulate the codebook design as a joint optimization problem and propose an iteration algorithm to obtain the enhanced codebook by combining the compressive sampling matching pursuit (CoSaMP) algorithm with the dictionary learning algorithm. In order to prove the validity of the proposed algorithm, we simulate and analyze the change of the spectral efficiency of the algorithm with the signal-to-noise ratio (SNR) and the number of radio frequency (RF) chains of different precoding schemes. The simulation results demonstrate that the spectral efficiency of the algorithm is obviously outstanding compared with that of the OMP-based joint codebook algorithm and the hybrid precoding algorithm with quantization algorithm under low SNR and different numbers of RF chains. Particularly, when SNR is lower than 0 dB, the proposed algorithm performs very close to the optimal unconstrained precoding algorithm.