The association between cannabis exposure and working memory impairment and its neural substrates remain unclear. In this cross-sectional observational study, we investigated this by examining the relationship between frequency of exposure to cannabis, working memory performance and regional brain volumes and tested whether lower volumes of cortical and subcortical structures mediate the association between cannabis exposure and working memory deficit using the Human Connectome Project data from 234 individuals with self-reported cannabis exposure and 174 individuals unexposed to cannabis. We tested the relationship between self-reported frequency of cannabis exposure and list-sorting working memory task performance (total number of correct responses), between T1 weighted MRI-derived regional grey-matter volumes and working memory task performance as well as between frequency of cannabis exposure and brain volumes after controlling for potential confounders. Finally, mediation analysis was carried out to test whether deficit in working memory performance associated with cannabis use was mediated by its association with lower grey-matter volume. Participants who reported higher frequency of cannabis use tended to have lower number of correct responses in the list-sorting working memory task and lower bilateral hippocampal volumes. Association between severity of cannabis exposure as indexed by frequency of cannabis use and impairment in working memory was mediated by lower left hippocampal volume in cannabis users. We report evidence in support of the left hippocampus volume-mediated working memory impairment associated with recreational cannabis exposure. Future studies employing prospective longitudinal design are necessary to examine the cause-effect relationships of cannabis exposure on working memory and brain volumes.