Semantic memory was evaluated in 124 epilepsy patients, including 84 with left ( n = 44) or right temporal lobe epilepsy (TLE) ( n = 40) and 40 with left ( n = 25) or right frontal lobe epilepsy (FLE) ( n = 15), in order to determine their verbal and visual deficits, and the neuroanatomical relationships between them. The controls were 35 healthy subjects. Semantic memory was assessed by means of Picture Naming, Picture Pointing, the verbal Pyramid and Palm Trees Test (PPTT), the visual PPTT, Object Decision Hard, and Drawing From Memory. Episodic memory was assessed by means of the Short Story, Rey's Complex Figure, the Verbal and Visual Selective Reminding Procedure and Brown–Peterson Procedure. Factor analysis of the epilepsy patients distinguished their semantic memory scores from other neuropsychological domains. The semantic memory factor was significantly related to the side of the epileptic region, with lower scores in the left hemisphere and left TLE patients. In comparison with the controls, the left TLE patients were significantly impaired on Picture Naming, Picture Pointing, and Object Decision Hard. Subsequent analyses showed that, in comparison with the controls and the right TLE patients, the left TLE patients with lateral temporal lobe lesions were impaired in Picture Naming whereas, in comparison with the controls, the left TLE patients with mesial temporal lobe lesions were impaired in Object Decision Hard. On the contrary, the episodic memory factor was not related to the side of the epileptic region, and a few material-specific tests revealed opposite impairments in the left and right hemisphere patients. These results show that left TLE may cause semantic memory deficits involving verbal and visual information. Unlike the material-specific pattern of episodic memory, this pattern of impairment is in line with the view of an amodal semantic store in which all of the information about a thing overlaps. The semantic memory impairment may reflect damage in the lateral and mesial temporal lobe regions that impair neocortical functions in storing and retrieving information or hippocampal functions in processing meaningful stimuli.