Semiconductor wastewater with high concentrations of fluoride and phosphate is an environmental issue that cannot be ignored. Moreover, the byproduct of recycled aggregates, concrete fines, cannot be reused in concrete manufacturing, which is a key issue to address for the sustainable development of the concrete industry. The objective of this study was to tackle the crucial environmental issues of these two industries by developing concrete fines as an alternative material to treat semiconductor wastewater. The chemical precipitation of calcium fluoride and hydroxyapatite in the presence of concrete fines was determined as the mechanism underpinning the removal of fluoride and phosphate in wastewater. Owing to the wide range of contaminant concentration and solution pH and the possibility of multi-stage treatment, the effects of the initial contaminant concentration (F: 100–1000 mg/L; P: 20–200 mg/L) and solution pH (pH: 2–7) on the removal reactions were determined. The highest F and P removal percentages were more than 99%, and the final F and P concentrations met the effluent standard (F: 15 mg/L, P: 1.3 mg/L). The removal reactions of F and P are generally in competition, and the removal of F has priority over the removal of P. The pseudo-second-order model can describe the kinetics of the removal reactions well. The formation of fluorapatite can reduce the F concentration below the concentration achievable by CaF2 precipitation alone. Furthermore, using the byproduct of recycled aggregates instead of conventional chemicals to treat semiconductor wastewater is promising in terms of reducing CO2 emissions, and prospective applications are discussed. This study can lead to the development of a sustainable and clean process for semiconductor wastewater treatment using byproducts from the concrete industry.