The reduced graphene oxide (rGO-oxalic) with rich carbonyl and hydroxyl groups was prepared by using oxalic acid as reducing agent, and the structure of rGO-oxalic was characterized by XRD, Raman, TEM, AFM, XPS and BET. The anion and cation in oxalic acid play an important role in the removal of oxygen groups on GO surface, but the chemical reaction and mechanism are different. The further study indicated that the GO was reduced rapidly and effectively in the initial stage of reaction, and it is selective reduction in the middle and final stage of the reaction, showing that more carbonyl and hydroxyl groups were retained. The supercapacitor with rGO-oxalic as electrode material exhibited the superior specific capacitance (301.1 F·g−1 and 217.0 F·g−1 at 1 A·g−1 in three- and two-electrode systems, respectively). Meanwhile, the capacitance retention was as high as 89.5% after 10,000 cycles at 5 A·g−1 for the symmetric supercapacitor. The research on the reduction mechanism of rGO-oxalic is beneficial to its potential application in energy storage and other fields.