We present the study of a proof-of-concept integrated device that can be used as a nonlinear broadband isolator. The device is based on the asymmetric loading of a highly-confining silicon-slot photonic coupler with graphene layers, whose ultrafast and low-threshold saturable absorption can be exploited for nonreciprocal transmission between the cross-ports of the coupler. The structure is essentially a non-Hermitian system, whose exceptional points are briefly discussed. The nonlinear device is modeled with a coupled Schrodinger equation system whose validity is checked by full-vector finite element-based beam-propagation method simulations in CW. The numerically computed performance reveals a nonreciprocal intensity range (NRIR) in the vicinity of 100 mW peak power with a bandwidth spanning tens of nanometers, from CW down to ps-long pulses. Finally, the combination of saturable absorption and self-phase modulation (Kerr effect) in graphene is studied, indicating the existence of two NRIR with opposite directionality.