This paper systematically studied the composition-controlled nonlinear optical properties and pulse modulation of ternary ReS2(1-x)Se2xalloys for the first time. The compositionally modulated characteristics of ReS2(1-x)Se2xon the band gap were simulated based on the first principles. We investigated the effect of the band gap on the saturable absorption properties. In addition, we demonstrated the modulation characteristics of different components ReS2(1-x)Se2xon 1.5μm Q-switched pulse performance. The Q-switched threshold, repetition rate, and pulse duration increase as the S(sulfur)-element composition rise. And pulse energy also was affected by the S(sulfur)-element composition. The ReS0.8Se1.2SA was selected to realize a conventional soliton with high energy in the all-fiber mode-locked laser. The pulse was centered at 1562.9 nm with a pulse duration of 2.26 ps, a repetition rate of 3.88 MHz, and maximum pulse energy of 1.95 nJ. This work suggests that ReS2(1-x)Se2xhas great potential in laser technology and nonlinear optics, and widely extends the material applications in ultrafast photonics.