It is important to find nontrivial constraint relations for color-ordered amplitudes in gauge theories. In the past several years, a pure group-theoretic iterative method has been proposed to derive linear constraints on color-ordered amplitudes in SU(N) gauge theories. In this paper, we use the same method to derive linear constraints on four-point gluon amplitudes in SO(N) and Sp(2N) gauge theories. These constraints are derived up to four-loop order. It is found that there are $n=1,6,10,13,16$ constraint relations at $L=0,1,2,3,4$ loop orders in both SO(N) and Sp(2N) cases. Correspondingly, the numbers of independent four-point color-ordered amplitudes are $2,3, 5, 8, 11$ at $L=0,1,2,3,4$ loop orders in both theories.