A robust and efficient quantum key agreement (QKA) protocol is presented with decoherence-free (DF) states and single-particle measurements. Compared with all the previous QKA protocols, which are designed in ideal condition, this protocol can not only guarantee both the security and fairness of the shared key but also be immune to collective decoherence. In addition, our protocol has a high intrinsic efficiency due to the utilization of the delayed measurement technique. Finally, we show that the proposed protocol is secure against the attacks from both outside eavesdroppers and inside dishonest participants.