Nitrogen incorporation changes the lattice spacing of SiC and can therefore lead to stress during physical vapor transport (PVT). The impact of the nitrogen-doping concentration during the initial phase of PVT growth of 4H-SiC was investigated using molten potassium hydroxide (KOH) etching, and the doping concentration and stress was detected by Raman spectroscopy. The change in the coefficient of thermal expansion (CTE) caused by the variation of nitrogen doping was implemented into a numerical model to quantitatively determine the stress induced during and after the crystal growth. Furthermore, the influence of mechanical stress related to the seed-mounting method was studied. To achieve this, four 100 mm diameter 4H-SiC crystals were grown with different nitrogen-doping distributions and seed-mounting strategies. It was found that the altered CTE plays a major role in the types and density of defect present in the grown crystal. While the mounting method led to increased stress in the initial seeding phase, the overall stress induced by inhomogeneous nitrogen doping is orders of magnitude higher.