Graphene is treated as an anisotropic material because it is one atom thick and its π electrons cause electric conduction in its plane. Based on the polarization-dependent absorption, we design compact broadband transverse-magnetic (TM)-pass polarizer by exploiting a graphene–silicon horizontal slot waveguide structure on silicon-on-insulator (SOI) platforms. The interaction between light and graphene is greatly enhanced by placing the double-layer graphene sheets adjacent to the slot waveguide. The selection of the geometric parameters (e.g., waveguide width, silicon, and Si3N4 thicknesses) is discussed. We study the mode properties, extinction ratio (ER), bandwidth, and insertion loss. The graphene–silicon slot waveguide based polarizer offers the performance of high extinction ratio, low insertion loss, broad bandwidth, small footprints, and compatibility with an SOI platform. By employing a 150 μ m long graphene–silicon horizontal slot waveguide, the ER is higher than 40 dB, and the insertion loss is less than 3 dB over the 1450–1650 nm wavelength range. We also analyze the impacts of practical fabrication imperfections on the operation performance, i.e., fabrication error tolerance, and give the possible fabrication process of graphene–silicon horizontal slot waveguide based polarizer.