It is very challenging to achieve direct capture of the circular polarization due to the complex mechanism of the chiral materials. We numerically demonstrate a circular polarization photodetector composed of the silicon (Si) metasurfaces rotated 45° clockwise presenting as the phase delayer and the grating-coupled quantum well infrared photodetectors (QWIP) chip detecting the transverse magnetic (TM) mode. The circular dichroism and extinction ratio of the circular polarization (CP) photodetector are 0.25 and 20 dB at 4 μm operation wavelength, respectively. In addition, the linear dichroism (LD) and extinction ratios (ER) of the linear polarization device are 0.22 and 60 dB, respectively. The excellent LD performance is caused by the Fano resonance of the Si/SiO2 linear grating corresponding to the interference of the bright electric dipole mode and the dark electric quadrupole mode. The full Stokes pixel of the six-image-element technique can almost accurately obtain arbitrary polarized light at 4 μm operation wavelength. The errors of the degree of linear polarizations (Dolp) and the degree of circular polarizations (Docp) are less than −30 dB and −20 dB, respectively.