We classify discrete modular symmetries in the effective action of Type IIB string on toroidal orientifolds with three-form fluxes, emphasizing on $T^6/\mathbb{Z}_2$ and $T^6/(\mathbb{Z}_2\times \mathbb{Z}_2^\prime)$ orientifold backgrounds. On the three-form flux background, the modular group is spontaneously broken down to its congruence subgroup whose pattern is severely constrained by a quantization of fluxes and tadpole cancellation conditions. We explicitly demonstrate that the congruence subgroups appearing in the effective action arise on magnetized D-branes wrapping certain cycles of tori.