In the Chern-Simons gauge theory on a manifold ${T}^{2}$\ifmmode\times\else\texttimes\fi{}${R}^{1}$ (two-torus\ifmmode\times\else\texttimes\fi{}time) the unitary operators, which induce large gauge transformations shifting the nonintegrable phases of the two dinstinct Wilson-line integrals on the torus by multiples of 2\ensuremath{\pi}, do not commute with each other unless the coefficient of the Chern-Simons term is quantized. In U(1) theory this condition gives the statistics phase \ensuremath{\theta}=\ensuremath{\pi}/n (n an integer). The condition coincides with the one previously derived on a manifold ${S}^{3}$ (three-sphere) for SU(N\ensuremath{\ge}3) theory but differs by a factor 2 for SU(2) theory. The requirement of the ${Z}_{N}$ invariance in pure SU(N) gauge theory imposes a stronger constraint.