In millimeter wave (mmWave) multiple-input multiple-output (MIMO) systems, the hybrid beamforming architecture has been put forward to reduce the high hardware cost and power consumption, which are resulted from the tremendous requirements of dedicated radio frequency (RF) chains. In this paper, we propose several strategies to design analog and digital precoders for a point-to-point (P2P) hybrid MIMO system. Aiming at minimizing the Euclidean distance between the optimal digital precoder and hybrid precoder, we decouple this matrix factorization problem into a nonconvex quadratically constrained quadratic programming (QCQP) problem and an unit-modulus least-squares (ULS) problem, which can be solved by the presented three alternating optimization algorithms. Simulation and analysis results indicate that the proposed semidefinite relaxation based alternating optimization (SDR-AO) algorithm can approach near-optimal spectral efficiency performance compared with previous algorithms in the literature, but shows extremely high computational complexity. The alternating direction method of multipliers based alternating optimization (ADMM-AO) algorithm is preferred in the case that the number of transmit antennas is much larger than that of receive antennas or the amount of data streams is small. Moreover, when equal number of RF chains and data streams are employed, the analytical constant modulus factorization based alternating optimization (ACMF-AO) algorithm is a better choice. Finally, the proposed algorithms can also be well applied in finite resolution phase shifters (PSs) of the analog component and are extended to wideband mmWave systems.