The photon counting detector based spectral CT system is attracting increasing attention in the CT field. However, the spectral CT is still premature in terms of both hardware and software. To reconstruct high quality spectral images from low-dose projections, an adaptive image reconstruction algorithm is proposed that assumes a known reference image (RI). The idea is motivated by the fact that the reconstructed images from different spectral channels are highly correlated. If a high quality image of the same object is known, it can be used to improve the low-dose reconstruction of each individual channel. This is implemented by maximizing the patch-wise correlation between the object image and the RI. Extensive numerical simulations and preclinical mouse study demonstrate the feasibility and merits of the proposed algorithm. It also performs well for truncated local projections, and the surrounding area of the region- of-interest (ROI) can be more accurately reconstructed. Furthermore, a method is introduced to adaptively choose the step length, making the algorithm more feasible and easier for applications.