Based on a data-driven approach, a computer-assisted workflow for the quantitative analysis of optical Kerr microscopy images of sintered FeNdB-type permanent magnets was developed. By analyzing the domain patterns visible in the Kerr image with data-driven approaches such as traditional machine learning and advanced deep learning, we can quantify grain orientation and size with a better trade-off between accuracy and higher throughput than electron backscatter diffraction (EBSD). The key distinction between traditional machine learning and advanced deep learning lies in feature extraction. Traditional methods require manual, user-dependent feature extraction from input data, while advanced deep learning achieves this automatically. The predictions from the trained models were compared to the measurements from EBSD for performance evaluation. The proposed data-driven model is trained on the dataset created from the correlative microscopy technique, which requires the images of grains extracted from the Kerr microscopy and corresponding EBSD grain orientation data (Euler angles). The fine-tuned deep learning model shows better generalization ability than the traditional machine learning models trained on the manually extracted features and resulted in a mean absolute error of less than 5° for grain orientation of the anisotropic magnet samples when evaluated against the measured EBSD values. The developed approach has reduced the measurement effort for grain orientation by 5 times and have sufficient accuracy when compared to the EBSD.Further, the application of the proposed approach to determine the quality of the alignment or texture in anisotropic sintered magnets and its relationship with the magnetic remanence based on reliable statistical grain orientation data has been discussed. This approach could emerge as a tool for rapidly analyzing large-scale samples to discover and quantify heterogeneities in grain size and grain orientation.