Hierarchical CuO@rGO nanostructures, consisting of CuO nanoparticles (NPs) and reduced graphene oxide (rGO) nanosheets, were prepared with a normal-pressure microwave-assisted in-situ growth process. In the CuO@rGO composites, CuO NPs with a size range of 4–11 nm are uniformly anchored on rGO surfaces. In contrast, CuO particles of large and non-uniform distribution in the CuO-doped rGO nanosheets (CuO/rGO) were prepared by a conventional hydrothermal method. The CuO@rGO sensors show high response and selectivity to H2S (1–10 ppm) at low operating temperatures (50–150 °C). The mass ratio of copper acetate to graphene oxide during preparation (6–10) is found to highly influence the gas-sensing properties, and the 8-CuO@rGO sample shows the highest H2S-sensing performance at 100 °C. Compared with pristine CuO nanocrystals, the 8-CuO@rGO composite exhibits considerably enhanced gas-sensing performance, such as good response, high selectivity and low optimal temperature. The improved H2S-sensing performance of the CuO@rGO composites can be attributed to the synergistic effect in components and their hierarchical nanostructure.