Oily sludge (OS) is a kind of hazardous waste generated from the petrochemical industry. Currently, pyrolysis has been widely applied for OS disposal, while low-oil content (<5 wt%) OS still lacks novel technology to achieve efficient resource utilization and harmful substances immobilization. In this study, a kind of OS-based geopolymer was developed by OS and ground granulated blast furnace slag (GGBS). The results showed that in geopolymer with 30 wt% OS, the content of total petroleum hydrocarbons (TPHs) decreased by 82%, Zn achieved 100% stabilization, and the 28 d compressive strength could still reach 32.8 MPa. The appropriate oil content filled the pores and cracks in geopolymer matrix. The constructed model compounds further elucidated the hydration mechanisms of OS-geopolymer. The nucleation effect of crude oil and micro-aggregate effect of minerals jointly improved the polymerization degree of C-(A)-S-H gels. OS promoted the transformation of [SiO4]4− monomers into C-(A)-S-H unbranched middle groups and three-dimensional networks, thereby efficiently stabilizing harmful substances. Sustainability analysis showed that OS-based geopolymer had good environmental and economic benefits. Overall, this work provides theoretical guidance for the green transformation of OS in the construction field.