With the remarkable growth of renewable energy resources, the grid-forming (GFM) inverter with the function of grid voltage/frequency support attracts much attention. Due to the inverter-grid interaction, the stability of the GFM inverter is a critical issue. The passivity-based analysis approach, which was widely applied to the conventional grid-following inverter, has been proved to be promising. Yet, its application to the GFM inverter is still insufficient. To this end, this paper conducts a comprehensive passivity-based analysis for the GFM inverter with single-loop voltage control. It finds that the two indices of the passivity-based stability criterion, i.e., the individual stability and the output impedance passivity, bring identical constraints on the voltage controller, and the passivity cannot be ensured with typical voltage controllers. To shrink the unexpected non-passive frequency ranges, a generic grid-current feedforward scheme is explored, and the proper feedforward functions compatible with different voltage controllers are derived. With the proposed scheme, the passivity can be guaranteed up to the Nyquist frequency. Finally, experimental results from a 6-kVA prototype are provided to verify the theoretical analysis.