Strong Kerr nonlinearity of graphene is a promising basis for the realization of all-optical nonlinear applications on a silicon chip. However, on one hand, the interaction of light with graphene as a two-dimensional material is weak and, on the other hand, the sign of the Kerr indices in silicon and graphene is different, which results in reverse index changes and consequently, reduces the total nonlinear index shift. In this paper, a slotted silicon nano-beam cavity is designed to both increase the interaction of light with the graphene sheet and simultaneously reduce the nonlinear phenomena in the silicon through the strong light confinement in the slot region and in the vicinity of the graphene layer. The designed cavity supports two TE modes, so it facilitates on-chip coupling of both the pump and signal to the cavity which is essential for final packaging. Optical amplitude modulation (or on / off switching) of the light in the device is theoretically investigated using the perturbation and the temporal coupled-mode theories. A modulation depth of >80% (∼8 dB) is obtained for the low pump power of 50 mW when pumped by a pulse with a 10-ps temporal full-width half-maximum and a 10-GHz repetition rate. The photonic bandwidth of the proposed structure is limited by the photon lifetime of the cavity and is as large as 123 GHz. The device is also highly compact with the overall footprint of ∼8.5 μm2. The combination of low-power, high-speed operation, large modulation depth, and the compactness of the proposed structure makes it a good candidate for the realization of all-optical integrated circuits.