Point-of-care testing (POCT) of chloramphenicol (CAP) in fresh milk from private ranches is essential to ensure food safety. It’s still a great challenge to meet simultaneously rapid and accurate during POCT of CAP. Here, we present a POCT device integrating hydrogel microsphere immunoassay and smartphone Fourier analysis for fast and sensitive detection of CAP. Based on the excellent biocompatibility and outstanding mechanical properties of hydrogel microspheres (100 μm), the antibody activity can be maintained and the mechanical deformation of the hydrogel cross-linking lattice is induced by squeezing to accelerate the antigen-antibody reaction of CAP and anti-CAP antibodies. The signal-to-noise ratio of the detection result is optimized adaptively by a designed microsphere Fourier analysis algorithm for precise CAP concentration detection. The results show that the method decreases incubation time from 60 min to 25 min and meets detection limit requirements of industrial production (0.05 ng/mL, with a dynamic range of 0.05–8 ng/mL). This method not only achieves rapid and accurate on-site detection of CAP in milk but also applies to other low concentrations of harmful substances detection in food field.