Several clinical trials are investigating the use of immune-targeted therapy with Programmed death ligand-1 (PD-L1) inhibitors for colorectal cancer (CRC), with promising results for patients with mismatch repair (MMR) deficiency or metastatic CRC. However, the prognostic significance of PD-L1 expression in CRC is controversial and such data are lacking in CRC from Middle Eastern ethnicity. We carried out this large retrospective study to investigate the prognostic and clinico-pathological impact of PD-L1 expression in Middle Eastern CRC using immunohistochemistry. A total of 1148 CRC were analyzed for PD-L1 expression. High PD-L1 expression was noted in 37.3% (428/1148) cases and was correlated with aggressive clinico-pathological features such as high malignancy grade (p < 0.0001), larger tumor size (p = 0.0007) and mucinous histology (p = 0.0005). Interestingly, PD-L1 expression was significantly higher in patients exhibiting MMR deficiency (p = 0.0169) and BRAF mutation (p = 0.0008). Furthermore, the expression of PD-L1 was found to be an independent marker for overall survival (HR = 1.45; 95% CI = 1.06–1.99; p = 0.0200). In conclusion, the results of this study indicate that PD-L1 expression could be a valid biomarker for poor prognosis in Middle Eastern CRC patients. This information can help in decision-making for anti-PD-L1 therapy in Middle Eastern CRC, especially for patients with MMR deficient tumors.