<sec>We report an experimental device and method of measuring the coercivity of soft magnetic material in an open magnetic circuit by using a pump-probe rubidium atomic magnetometer. The experimental device consists of a background magnetic field generation module, a pump-probe atomic magnetometer, a soft magnetic material magnetization and demagnetization module, and a software in a computer. The uniform background magnetic field ranging from 200 nT to 20000 nT along the <i>z</i>-axis at the rubidium bubble’s position is generated by a current carrying coil which is placed in a five-layer magnetic shielding cylinder. The saturation magnetization and demagnetization of soft magnetic material are realized by the soft magnetic sample magnetization and demagnetization module, respectively, which consists of a sample chamber, a soft magnetic sample, a magnetizing coil, a precision current source and a sample transfer rod. The sample chamber is placed in the magnetic field uniform area of the magnetizing coil which is placed in the magnetic shielding cylinder, and the sample transfer rod is used to transfer the soft magnetic sample into the center of the magnetizing coil. Both the rubidium bubble and the soft magnetic sample are placed on the <i>z</i>-axis of the magnetic shielding cylinder, and their distance is greater than or equal to 10 cm. The axis of the magnetizing coil coincides with the axis of the magnetic shielding cylinder, which ensures that the background magnetic field, the magnetic field generated by the magnetizing coil, and the magnetic field generated by the soft magnetic sample at the rubidium bubble’s position are all parallel to the axis direction of the magnetic shielding cylinder. The software in the computer realizes the magnetization and demagnetization of the soft magnetic sample by controlling the current output by the precision current source to the magnetizing coil, and also controls the pump-probe atomic magnetometer to measure the magnetic field at the rubidium bubble’s position. When the soft magnetic sample is magnetized or demagnetized in cycles, the magnetic field generated by the soft magnetic sample at the rubidium bubble’s position is obtained by subtracting the magnetic field value measured in advance when the sample is not placed in the sample chamber from the magnetic field value measured at same current value when the sample is placed in the sample chamber. Note that the sample does not move during the hysteresis loop measurement. When the magnetization of the soft magnetic sample decreases from the saturation value to zero, the magnetic field generated by the soft magnetic sample at the rubidium bubble’s position is zero, and the average coercivity of the sample can be calculated from the hysteresis loop.</sec><sec>We use a superconducting quantum interference device (SQUID), a pump-detection rubidium atomic magnetometer and a Hall probe to measure the coercivity of the same permalloy strip sample, the average coercivities are 42.15 A/m, 40.632 A/m and 38.64 A/m, the biases of the hysteresis loops are 229.74 A/m, –0.95 A/m and –52.88 A/m, and the times of each measurement cycle are 9639 s, 1144 s, and 1630 s, respectively. The reproducibility of ten repeated measurements by using the pump-probe atomic magnetometer, expressed as relative standard deviation, is 0.16%, which is an order of magnitude higher than the counterparts from the methods described in China’s national standards GB/T 3656-2008 and GB/T 13888-2009. The accumulated drift of the biases of ten hysteresis loops measured by the pump-probe atomic magnetometer is 0.3 A/m. Based on the above experimental results, the coercivity measurement method by using the pump-probe atomic magnetometer has the advantages of no zero-point drift, good repeatability, fast measurement speed, and <i>in-situ</i> measurement, and has the potential applications in the basic research field and industrial field of magnetism.</sec>