Rheumatoid arthritis is an autoimmune disease that causes joint damage due to inflammation in the soft tissue lining the joints known as the synovium. It is vital to identify joint damage as soon as possible to provide necessary treatment early and prevent further damage to the bone structures. Radiographs are often used to assess the extent of the joint damage. Currently, the scoring of joint damage from the radiograph takes expertise, effort, and time. Joint damage associated with rheumatoid arthritis is also not quantitated in clinical practice and subjective descriptors are used. In this work, a description of a pipeline of deep learning models to automatically identify and score rheumatoid arthritic joint damage from a radiographic image is provided. An automatic tool was built to produce scores with extremely high balanced accuracy within a couple of minutes and utilizing this would remove the subjectivity of the scores between human reviewers. Using a joint segmentation approach and training joint score prediction models with ordinal class encoding, under-sampling, and transfer learning, the joint wised ±1 balanced accuracies ranging from 91.51% to 97.30% were achieved. The ±1 balanced accuracy of the 4 models showed great potential in achieving industry-standard reliability.