In this paper, we consider a cooperative communication system with differential modulation and relay-selection (DM---RS) in multi-relay networks, where the best relay is selected to forward the source node's signal to the destination node with detect-and-forward (DetF) protocol. Unlike the conventional DM---RS---DetF scheme using phase shift keying (PSK), we propose a DM---RS---DetF scheme with quadrature amplitude modulation (QAM). Since the differential PSK scheme cannot be applied to the QAM constellations directly, we firstly develop the modulation and demodulation methods for the differential QAM scheme. Then, we derive a closed-form approximation and an upper bound of the symbol error rate for the DQAM---RS---DetF scheme over independent Rayleigh fading channels, by using the approximated signal-to-noise ratio of the equivalent relay link. Simulations results verify the validity of the analytical results and further show that when the modulation order $$M>8$$ in Rayleigh fading channels, the DQAM---RS---DetF scheme outperforms the DPSK---RS---DetF scheme.