Objectives: Endoscopic severity of ulcerative colitis predicts clinical outcomes, making it critical for assessing efficacy of new therapies. However, its use in routine practice is limited by availability of experienced and trained human reviewers. We investigated the feasibility of using deep learning algorithms to independently and accurately grade severity of ulcerative colitis from still images and full motion video colonoscopies. Setting: Patients with ulcerative colitis undergoing colonoscopy at a single tertiary care referral center in the United States between 2007-2017. Design: Retrospective grading of ulcerative colitis severity was performed on colonoscopic images using the standardized Mayo score by at least 2 human reviewers as the gold standard for diagnosis. A 159-layer convolutional neural network was employed as a deep learning algorithm for feature extraction and used to train and categorize images into two clinically-relevant groups: remission (defined as Mayo 0-1) and moderate-to-severe disease (defined as Mayo 2-3). Seventy-percent of the cohort was used to train the algorithm and 30% of the cohort was used to test it. An independent set of graded full motion video colonoscopies (N=30) unseen by the algorithm was then used for external validation. Main Outcome Measure: Performance of the deep learning automated grading system was assessed using Area Under the Receive Operating Curve (AuROC), positive (PPV) and negative predictive values (NPV),and kappa statistics for agreement relative to the gold standard of human reviewers. 14,830 endoscopic still images over 3,038 unique individuals with ulcerative colitis underwent grading. The accuracy of the convolutional neural network approach for distinguishing endoscopic remission from moderate-severe disease was excellent at 93.7% (95% CI 96.4 to 97.9) with an AuROC of 0.979, yielding a PPV of 88.41% and NPV of 93.75%. Misclassification of exact Mayo endoscopic severity score was more common for intermediate endoscopic severities (Mayo grade 1 and 2), with an overall weighted Kappa agreement of k=0.855 (95% CI 0.847 to 0.863) between deep learning prediction and human reference score. Applying score prediction methods to unseen, full-motion colonoscopy videos demonstrated similar accuracy at identifying endoscopic remission (AuROC 0.981) and agreement of scores with human reference scores (k=0.821, 95% CI 0.803 to 0.839, Table 1). A deep learning algorithm accurately grades severity of ulcerative colitis as compared with experienced human reviewers. Automated analysis of colonoscopy videos could substantially improve access to standardized assessments of ulcerative colitis in routine practice and research.