The plasmon resonance of a structure is primarily dictated by its optical properties and geometry, which can be modified to enable hot-carrier photodetectors with superior performance. Recently, metal alloys have played a prominent role in tuning the resonance of plasmonic structures through chemical composition engineering. However, it has been unclear how alloying modifies the time dynamics of the generated hot-carriers. In this work, we elucidate the role of chemical composition on the relaxation time of hot-carriers for the archetypal AuxAg1-x thin film system. Through time-resolved optical spectroscopy measurements in the visible wavelength range, we measure composition-dependent relaxation times that vary up to 8× for constant pump fluency. Surprisingly, we find that the addition of 2% of Ag into Au films can increase the hot-carrier lifetime by approximately 35% under fixed fluence, as a result of a decrease in optical loss. Further, the relaxation time is found to be inversely proportional to the imaginary part of the permittivity. Our results indicate that alloying is a promising approach to effectively control hot-carrier relaxation time in metals.