The relationship between the grain size of as-cast and cold rolled 16%Cr ferritic stainless steel and the surface roughness defect, called ridging during forming was investigated. The ridging height corresponded to the grain size of the as-cast sample. The nitrogen content of 140 ppm yielded the minimum grain size and the minimum ridging height observed, whereas the nitrogen content of 50 ppm yielded the maximum grain size and the maximum ridging height observed. Ridging results from different plastic anisotropies of band structure composed of colonies. Through the EBSD analysis, the texture of mixed colonies composed of ND//{112} and ND//{331} in the 50 ppm nitrogen steel underwent more severe ridging than the randomly texture in the 140 ppm nitrogen steel sample. Therefore, an effective means to reduce the ridging of ferritic stainless steel during the forming process is to form a random texture by enhancing the formation of fine equiaxed grain during the casting process. During equal holding times at 1200 °C, the 80 ppm nitrogen sample was definitely coarsened, whereas the 200 ppm nitrogen sample underwent slower grain growth. Zener pinning force, which is proportional to the number of TiN particles on grain boundaries, was relatively strong in samples of 200 ppm nitrogen content, corresponding to slower grain growth. Although the Zener pinning force great affected with increasing nitrogen content, there may not affect the trend of initial cast grain size to be changed as much during annealing.