SWI/SNF complexes represent a family of multi-subunit chromatin remodelers that are affected by alterations in >20% of human tumors. While mutations of SWI/SNF genes are relatively uncommon in prostate cancer (PCa), the literature suggests that deregulation of various subunits plays a role in prostate tumorigenesis. To assess SWI/SNF functions in a clinical context, we studied the mutually exclusive, paralogue accessory subunits SMARCD1, SMARCD2, and SMARCD3 that are included in every known complex and are sought to confer specificity. Performing immunohistochemistry (IHC), the protein levels of the SMARCD family members were measured using a tissue microarray (TMA) comprising malignant samples and matching healthy tissue of non-metastatic PCa patients (n = 168). Moreover, IHC was performed in castration-resistant tumors (n = 9) and lymph node metastases (n = 22). To assess their potential role as molecular biomarkers, SMARCD1 and SMARCD3 protein levels were correlated with clinical parameters such as T stage, Gleason score, biochemical recurrence, and progression-free survival. SMARCD1 protein levels in non-metastatic primary tumors, lymph node metastases, and castration-resistant samples were significantly higher than in benign tissues. Likewise, SMARCD3 protein expression was elevated in tumor tissue and especially lymph node metastases compared to benign samples. While SMARCD1 levels in primary tumors did not exhibit significant associations with any of the tested clinical parameters, SMARCD3 exhibited an inverse correlation with pre-operative PSA levels. Moreover, low SMARCD3 expression was associated with progression to metastasis. In congruence with previous literature, our results implicate that both SMARCD1 and SMARCD3 may exhibit relevant functions in the context of prostate tumorigenesis. Moreover, our approach suggests a potential role of SMARCD3 as a novel prognostic marker in clinically non-metastatic PCa.