We reported the magnetic and dielectric properties of the polar magnet CaBaCo4-xMnxO7 and CaBaCo4-xNixO7 (x = 0.04, 0.08). The result shows that Mn ions exist in the sample in + 2 and + 3 valence, replacing both Co2+ and Co3+, and the Ni ions exist in + 2 valence, only replacing Co2+. Mn ion doping does not change the ratio of Co2+ to Co3+ in the crystal, leading to similar magnetic behavior to CaBaCo4O7. Ni ions increase the relative content of Co3+, which has a greater damage to the ferromagnetic interaction. Meanwhile, Mn-doping can partially release the magnetic frustration and cause the spin freezing temperature shifting to high temperature. It also slightly weakened the FIM interaction in the sample. While Ni-doping enhances the antiferromagnetic interaction and induces an additional antiferromagnetic phase at 82 K in the sample. The dielectric behavior of all samples is obviously abnormal at magnetic phase transitions, which proves the magnetodielectric effect of the samples. After Mn and Ni ions doping, the magnetodielectric properties of the samples are improved. Compared with Mn ions, Ni ions have a greater effect on the dielectric properties of CaBaCo4O7.