Low-dimensional photoconductors have extraordinarily high photoresponse and gain, which can be modulated by gate voltages as shown in literature. However, the physics of gate modulation remains elusive. In this work, the physics of gate modulation in silicon nanowire photoconductors with the analytical photoresponse equations is investigated. It is found that the impact of gate voltage varies vastly for nanowires with different size. For the wide nanowires that cannot be pinched off by high gate voltage, it is found that the photoresponses are enhanced by at least one order of magnitude due to the gate-induced electric passivation. For narrow nanowires that starts with a pinched-off channel, the gate voltage has no electric passivation effect but increases the potential barrier between source and drain, resulting in a decrease in dark and photocurrent. For the nanowires with an intermediate size, the channel is continuous but can be pinched off by a high gate voltage. The photoresponsivity and photodetectivity is maximized during the transition from the continuous channel to the pinched-off one. This work provides important insights on how to design high-performance photoconductors.