Neutralizing antibodies (nAbs) produced from infection or vaccination play an important role in acquired immunity. Determining virus-specific nAb titers is a useful tool for measuring aquired immunity in an individual. The standard methods to do so rely on titrating serum samples against live virus and monitoring viral infection in cultured cells which requires high biosafety level containment. The surrogate virus neutralization test (sVNT) reduces the biohazards and it is suitable for designing rapid test device in a lateral flow assay (LFA) format. Here, we introduce the fabrication and development of a unique paper-based LFA device for determining the level of SARS-CoV-2 nAb in a sample with a semiquantitative direct colorimetric readout. A LFA-based gradient assay design was used to facilitate the sVNT, where the spike glycoprotein receptor binding domain (RBD) and angiotensin-converting enzyme 2 (ACE2) stand in as proxies for viruses and cells, respectively. The gradient assay employed multiple test dots of ACE2 spotted in increasing concentration along the sample flow path and gold nanoparticle-conjugated RBD for readout. In this way, the number of developed spots is inversely proportional to the concentration of nAbs present in the sample. The assay was tested with both standard solutions of nAb as well as human serum samples. We have demonstrated that the device can effectively provide semiquantitative test results of nAbs by direct instrument-free colorimetric detection.