In recent years, nitrogen vacancy (NV) centers have shown great potential for application in quantum sensor research due to their high sensitivity to magnetic fields and excellent spatial resolution. However, in practical applications, it is necessary to eliminate environmental requirements and bulky instruments brought by the optical detection of magnetic resonance (ODMR) measurement method.. In this paper, we propose an efficient portable magnetometer with a miniature confocal system (CSPM) to resist background noise and improve integration. The confocal system can automatically focus, and the volume is limited to 11 cm3. Moreover, a fiber optic flange, an antenna and a homemade integrated photodetector are integrated in the magnetometer to achieve the actual sensitivity of sub-40 nT/Hz1/2. In the experimental process of optimizing system parameters, we discovered the important influence of beam focusing or microwave uniform operation on the detection signal and proposed specific optimization methods, which have guiding significance for future integrated design. Through comparative experiments with an integration scheme that does not use uniform operation, we found that the CSPM reduces the system noise by 4.2 times and that the signal-to-noise ratio (SNR) increases by 6.2 dB.