We investigate the interaction effect between oxygen impurities in crystalline germanium on the basis of a quantum rotor model. The dipolar interaction of nearby oxygen impurities engenders non-trivial low-lying excitations, giving rise to anomalous behaviors for oxygen-doped germanium (Ge:O) below a few degrees Kelvin. In particular, it is theoretically predicted that Ge:O samples with oxygen-concentration of 10$^{17-18}$cm$^{-3}$ show (i) power-law specific heats below 0.1 K, and (ii) a peculiar hump in dielectric susceptibilities around 1 K. We present an interpretation for the power-law specific heats, which is based on the picture of local double-well potentials randomly distributed in Ge:O samples.