Waste concrete is one of the largest contributors to solid waste, but its high value-added utilization is still shortage of economic and effective ways. In this study, the Ca-rich cement mortar (CCM) developed from waste concrete was innovatively used to remove phosphorus from wastewater. The aims of this study were to identify the phosphorus removal mechanisms of CCM and determine its phosphorus removal capacity. Also, the results were used to evaluate the potential use of the CCM as an alternative material to reduce phosphorus pollution. The results show that the Ca2+ releasing capacity of the CCMs modified by heat treatment (800 °C for 20 min) or hydrochloric acid (5.0 mmol HCl for per gram of CCM) are significantly enhanced because of the formation of porous microstructure and highly soluble calcareous compound. The Ca2+ released combined with phosphorus under an alkaline condition (pH > 10) to form Ca5(PO4)3(OH) precipitate, which was bound on the porous surface of the CCMs particles. The phosphorus binding capacity of modified CCM reached 100 mg P/g, and the phosphate concentration in wastewater can be reduced from 100 mg/L to below 0.1 mg/L within 30 min. The Ca2+ releasing process form CCM can be divided into two stages: a primarily drastic increasing period and a subsequent stable stage; and it can be described by pseudo-second-order kinetic model. The modified CCM developed from waste concrete blocks exhibited sufficient potential for phosphorus removal/recovery.