The effect of nitrogen on the growth of boron-doped diamonds was investigated by removing or adding nitrogen impurities. Optical microscopy images showed that adding a small amount of boron to nitrogen-free diamond completely transformed the diamond into an opaque black color. In the presence of small amounts of boron, the addition of nitrogen diminished the chromogenic properties of boron impurities in diamond. The FTIR spectra showed a compensatory interaction between boron and nitrogen in diamond, causing a portion of the nitrogen to exist as N+ center. Raman spectroscopy confirmed that adding small amounts of nitrogen to diamond reduced the stresses in the diamond and improved its quality, whereas adding excessive amounts of nitrogen reduced the quality. The Hall effect measurements showed that adding nitrogen to boron-doped diamond reduced its p-conductivity, causing an increase in its resistivity and a decrease in its carrier concentration.