The step-by-step annealing of natural plastically deformed type IaAB diamond revealed new information on the N1 center, which is two nitrogen atoms separated by one carbon in an ionized state (S = 1/2). A slight increase in the intensity of the N1 electron paramagnetic resonance (EPR) spectrum after annealing at 1000 °C (10h) was observed, and the complete disappearance was detected at a temperature of 1300 °C (4 h). This behavior of the N1 paramagnetic center was associated with the annealing of electron acceptors, and we analyzed the conditions for the formation of complex vacancy defects. The temperature of 1000 °C corresponds to the formation of three vacancy chains from single vacancies and 1300 °C corresponds to their disappearance. We experimented with electron irradiation of the crystals (energy of 3 MeV and a fluence of 1018 e/cm2) followed by annealing at 1000 °C (for 10 h) and the N1 EPR spectrum appeared back with an intensity five times higher than in the initial crystal. The next heat treatment at 1400 °C (2 h) resulted in the expected vanish of the N1 EPR spectrum. In this case, electron irradiation followed by annealing at 1000 °C led to the restoration of a very weak EPR spectrum. The presence of the “EPR-silent” N1 centers in a neutral state in plastically deformed diamond prompted us to investigate their existence in crystals that were not subjected to plastic deformation. We generated acceptor centers in different diamonds by the above method and found N1 centers in the type IaAB natural and type IaA synthetic diamonds. The results obtained can suggest that the rate-limiting step in the aggregation of two impurity nitrogen atoms into close pair is the neutral N1 defect. In the ionized state of N1, the Coulomb repulsion between the unpaired extra electrons of nitrogen is eliminated, reducing the energy barrier in the final stage of transformation into the A center.