Limited by the number of illumination fields and the speed of a spatial light modulator, single-pixel imaging (SPI) cannot realize real-time imaging and fast classification of an object. In this paper, we proposed the circular harmonic Fourier single-pixel imaging (CHF-SPI) for the first time to realize fast imaging and classification of objects. The light field distribution satisfies the circular harmonic Fourier formula, and the light intensity values of the single-pixel detector are equivalent to the circular harmonic Fourier moments. Then the target can be reconstructed under low sampling ratio by inverse transformation. Through simulation and experimental verification, clear imaging can be performed at a sampling ratio of 0.9%. In addition, circular harmonic Fourier moments are used to construct multi-distortion invariant to classify objects with rotation and scale change. The scale change multiples of objects can be calculated and the objects can be classified by using 10 light fields. It is of great significance to classify objects quickly without imaging.