The use of Millimeter-wave (mm-Wave) has recently immensely enhanced in various communication applications due to massive technological developments in wireless communications. Furthermore, mm-Wave consists of a high bandwidth spectrum that can handle large demands of data transmission and internet services. However, high interference is observed in previous research at the time of spectrum handover from secondary (unlicensed) users to primary (licensed) users. Thus, interference reduction by achieving high spectral efficiency and an easy spectrum handoff process with minimum delay is an important research area. Therefore, a Hybrid Pre-coder Design based Spectrum Handoff (HPDSH) Algorithm is proposed in this article to increase spectrum efficiency in Cognitive Radio Networks(CRNs) and to access the large bandwidth spectrum of mm-Wave systems to meet the high data rate demands of current cellular networks. Moreover, a HPDSH Algorithm is presented to enhance spectral efficiency and this algorithm is utilized to take handover decisions and select backup channels. Here, different scenarios and parameters are considered to evaluate the performance efficiency of the proposed HPDSH Algorithm in terms of spectral efficiency and Signal to Noise (SNR) ratio. The proposed HPDSH Algorithm is compared against precoding methods like the OMP algorithm and SIC based methods.