An unmanned helicopter is one of the main platforms for conducting unmanned aerial vehicle aeromagnetic measurements and combines the advantages of rotary-wing and fixed-wing unmanned aerial vehicles. However, unmanned helicopter-based aeromagnetic measurement systems face complex static magnetic noise and maneuvering magnetic interference, which limit their practical performance. To address this issue, an improved multi-channel frequency measurement algorithm for the optically pumped magnetic sensor is proposed to suppress the static magnetic noise proportional to the frequency noise generated by the random quantization error and the airborne electromagnetic interference. A novel aeromagnetic compensation method for the maneuvering magnetic interference is then proposed to weaken the negative effects of the strong multicollinearity of the attitude parameters of the unmanned helicopter on the compensation accuracy and stability by introducing a regularization term and weight matrix. In addition, dedicated software is developed for the real-time calculation and compensation of magnetic interference fields. A dedicated unmanned-helicopter-based aeromagnetic measurement system is developed, and ground and flight experiments are carried out. The ground test results indicate that the static noise of the proposed system is only 0.000 82 nT. In the flight experiments, the system achieves an improvement ratio of 8.33, which is higher than the improvement ratio of 4.37 for a state-of-the-art commercial compensator. Furthermore, the dynamic noise after compensation decreases by 37.6% from 0.0157 to 0.0098 nT.