Grp94 is the endoplasmic reticulum paralog of the hsp90 family of chaperones, which have been targeted for therapeutic intervention via their highly conserved ATP binding sites. The design of paralog-selective inhibitors relies on understanding the protein structural elements that drive higher affinity in selective inhibitors. Here, we determined the structures of Grp94 and Hsp90 in complex with the Grp94-selective inhibitor PU-H36, and of Grp94 with the non-selective inhibitor PU-H71. In Grp94, PU-H36 derives its higher affinity by utilizing Site 2, a Grp94-specific side pocket adjoining the ATP binding cavity, but in Hsp90 PU-H36 occupies Site 1, a side pocket that is accessible in all paralogs with which it makes lower affinity interactions. The structure of Grp94 in complex with PU-H71 shows only Site 1 binding. While changes in the conformation of helices 4 and 5 in the N-terminal domain occur when ligands bind to Site 1 of both Hsp90 and Grp94, large conformational shifts that also involve helix 1 are associated with the engagement of the Site 2 pocket in Grp94 only. Site 2 in Hsp90 is blocked and its helix 1 conformation is insensitive to ligand binding. To understand the role of helix 1 in ligand selectivity, we tested the binding of PU-H36 and other Grp94-selective ligands to chimeric Grp94/Hsp90 constructs. These studies show that helix 1 is the major determinant of selectivity for Site 2 targeted ligands and also influences the rate of ATPase activity in Hsp90 paralogs.