A series of Al-doped CoP2/rGO composites have been fabricated by in-situ precipitation and phosphorization protocol. The well-ordered arrangement and widely adjustable mole ratio of cations within Co–Al layered double hydroxide (CoAl-LDH) make it suitable as a precursor for phosphiding and generating evenly dispersed Al element within CoP2 structure. The effect of Al doping content on the catalytic activities for hydrogen evolution reaction thus could be discussed in detail. Electrocatalytic performance indicates that the doping of Al element within CoP2 structure could efficiently prevent the oxidation of phosphide on the surface, and thus resulting in the high electrocatalytic activity for hydrogen evolution reaction (HER). Among all the Al-doped CoP2/rGO materials, the composite with the Co/Al mole ratio of 2/1 presents the highest HER activity and stability whatever under acidic or basic reaction conditions, which could be comparable to the most reported HER electrocatalysts and even the commercial 20% Pt/C. At the current density of −10 mA cm−2 in 0.5 M H2SO4, the overpotential is only 59 mV with a Tafel slope of 73.2 mV dec−1. Interestingly, the Al content could be dissolved under basic media and resulting in a porous structured electrode. Only the overpotential of 60 mV is needed to get the current density of −10 mA cm−2 in 1 M KOH.