We propose a protocol to direct implement the two-qubit controlled phase gate with cross-Kerr nonlinearity by using the interference of polarized photons. The protocol is based on optical elements, single polarization photons, cross-Kerr nonlinearity and the conventional photon detectors, which are feasible with existing experimental technology. Compared with Zou's protocol (Phys. Rev. 2007 A 75 034302), this protocol is to replace the entangled state resources used by Zou with much simpler single-photon resources and has a higher success probability. All these advantages make this protocol more efficient and more convenient than others in the applications in quantum communication.