The molybdenum silicon nitride (Mo–Si–N) films were deposited by a radio frequency (RF) magnetron reactive dual-gun co-sputtering technique with process control on input power and gas ratio. Composition variation, microstructure evolution, and related mechanical and tribological behavior of the Mo–Si–N coatings were investigated. The N2/(Ar + N2) flow ratios were controlled at 10/20 and 5/20 levels with the tuning of input power on the Si target at 0, 100, and 150 W. As the silicon contents increased from 0 to 33.7 at.%, the film microstructure evolved from a crystalline structure with Mo2N and MoN phases to an amorphous feature with the Si3N4 phase. The analysis of selected area electron diffraction patterns in TEM also indicated an amorphous feature of the Mo–Si–N films when Si content reached 20 at.% and beyond. The hardness and Young’s modulus changed from 16.5 to 26.9 and 208 to 273 GPa according to their microstructure features. The highest hardness and modulus were attributed to nanocrystalline Mo2N and MoN with Si solid-solution. The crystalline Mo–Si–N films showed a smooth tribological track and less wear failure was found. In contrast, the wear track with severe failures were observed for Mo–N and amorphous Mo–Si–N coatings due to their lower hardness. The ratios of H/E and H3/E2 were intensively discussed and correlated to the wear behavior of the Mo–Si–N coatings.