Quasi-bound states in the continuum (quasi-BIC) have tunable high Q factors and thus have many potential applications in optoelectronics. Here, we propose to construct a quasi-BIC with a perturbated elliptical-cylinder photonic crystal slab, and then develop a new way based on the quasi-BIC to address the background noise problem of infrared detectors. By integrating the photonic crystal slab with a quantum well infrared photodetector (QWIP), and by pushing the quasi-BIC system with absorption to the critical coupling state, an ultra-narrow photoresponse band (Q factor equal to 1150) with a high peak quantum efficiency (over 60%) is achieved in the long-wavelength infrared range. The critical coupling state is realized by matching the radiation Q factor, which is mainly controlled by the quasi-BIC, to the absorption Q factor, which is mainly controlled by the doping of the quantum wells (QWs). This device rejects most background radiation and thus significantly suppresses the background noise, resulting in a background limited specific detectivity (DBLIP∗) 11.85 times that of a conventional ideal photoconductor. A detailed formula is employed for calculation of DBLIP∗, incorporating the wavelength and incident angle dependence of the quantum efficiency. The ultra-narrow photoresponse band with a high peak responsivity can be tuned over the photosensitive wavelength range of the QWs by simply modifying the in-plane structural parameters. This work opens a new avenue to greatly enhance the specific detectivity for infrared detectors based on the joint effect of quasi-BIC and critical coupling.