Earth-abundant transition metal dichalcogenide nanosheets have emerged as an excellent catalyst for electrochemical water splitting to generate H2. Alloying the nanosheets with heteroatoms is a promising strategy to enhance their catalytic performance. Herein, we synthesized hexagonal (2H) phase Mo1-xNbxSe2 nanosheets over the whole composition range using a solvothermal reaction. Alloying results in a variety of atomic-scale crystal defects such as Se vacancies, metal vacancies, and adatoms. The defect content is maximized when x approaches 0.5. Detailed structure analysis revealed that the NbSe2 bonding structures in the alloy phase are more disordered than the MoSe2 ones. Compared to MoSe2 and NbSe2, Mo0.5Nb0.5Se2 exhibits much higher electrocatalytic performance for hydrogen evolution reaction. First-principles calculation was performed for the formation energy in the models for vacancies and adatoms, supporting that the alloy phase has more defects than either NbSe2 or MoSe2. The calculation predicted that the separated NbSe2 domain at x = 0.5 favors the concurrent formation of Nb/Se vacancies and adatoms in a highly cooperative way. Moreover, the Gibbs free energy along the reaction path suggests that the enhanced HER performance of alloy nanosheets originates from the higher concentration of defects that favor H atom adsorption.