Language comprehension involves both sub-lexical (e.g., phonological) and lexical-semantic processing. We conducted a task using functional magnetic resonance imaging (fMRI) to compare the processing of verbs in these two domains. Additionally, we examined the representation of concrete-motor and abstract-non-motor concepts by including two semantic categories of verbs: motor and mental. The findings indicate that sub-lexical processing during the reading of pseudo-verbs primarily involves the left dorsal stream of the perisylvian network, while lexical-semantic representation during the reading of verbs predominantly engages the ventral stream. According to the embodied or grounded cognition approach, modality-specific mechanisms (such as sensory-motor systems) and the well-established multimodal left perisylvian network contribute to the semantic representation of both concrete and abstract verbs. Our study identified the visual system as a preferential modality-specific system for abstract-mental verbs, which exhibited functional connectivity with the right crus I/lobule VI of the cerebellum. Taken together, these results confirm the dissociation between sub-lexical and lexical-semantic processing and provide neurobiological evidence of functional coupling between specific visual modality regions and the right cerebellum, forming a network that supports the semantic representation of abstract concepts. Further, the results shed light on the underlying mechanisms of semantic processing and contribute to our understanding of how the brain processes abstract concepts.