Studies have revealed that memory performance can be affected by perceived gaze direction. However, it remains unclear whether direct gaze promotes or hinders word memory, and the effect of gaze direction on memory of words with different concreteness requires investigation. In the study phase, concrete and abstract words were presented on direct- or averted-gaze faces, and participants were instructed to judge gaze direction and memorize words. In the test phase, participants were asked to discriminate whether a word was old or new. Electroencephalogram recordings were taken in both phases. Behavioral and time-frequency results verified the direct-gaze memory advantage, showing that memory performance was better in the direct-gaze condition than the averted-gaze condition for both concrete and abstract words. Event-related potential results showed that in both direct- and averted-gaze conditions, the early old/new effects (FN400) associated with familiarity were only elicited for concrete words but not abstract words. The late old/new effects (LPC) associated with recollection were elicited in all conditions. More importantly, concrete words elicited greater LPC than abstract words in the direct-gaze condition, whereas there was no such significant LPC difference in the averted-gaze condition. Topographic map analysis found that neural generators between concrete and abstract words differed in the direct-gaze condition but not in the averted-gaze condition. The study supports the hypothesis that direct-gaze promotes memory performance. Furthermore, it is mainly in memory recollection that gaze direction affects words with different concreteness.