Orthographic processing is crucial in reading. For the Chinese language, sub-lexical processing has already taken place at radical level. Previous literature reported early position-specific radical representations and later position-general radical representations, implying a possible separating process of abstract position information irrespective of radicals per se from radical representations during orthographic processing. However, it remains largely unclear whether the abstract pattern of spatial arrangement of radicals can be rapidly extracted, and if so, whether this extraction takes place at the visual cortex, the very first processing center. As the visual cortex is documented to actively participate in orthographic processing, it may also play a role in the possible extraction of abstract orthographic patterns of Chinese characters. Hence, we hypothesize that abstract orthographic patterns of Chinese characters are covertly extracted at the visual cortex during reading. In this study, we investigated whether the visual cortex could rapidly extract abstract structural patterns of Chinese characters, using the event-related potential (ERP) technique. We adopted an active oddball paradigm with two types of deviant stimuli different only in one feature, structural or tonal, from standard stimuli; in each of the two sessions, subjects focused conscious attention on one feature and neglected the other. We observed that the ERPs recorded at occipital electrodes responded differentially to standard and structural deviant stimuli in both sessions, especially within the time range of the occipital P200 component. Then, we extracted three source waves arising from different levels of the visual cortex. Early response differences (from 88 to 456 ms after stimulus onset) were observed between the source waves, probably arising from left primary/secondary and bilateral associative visual cortices, in response to standard and deviant stimuli that violated abstract structural patterns, whether subjects focused their attention on the character structure or not. This suggests rapid extraction of abstract structural patterns of Chinese characters in the visual cortex, no matter the abstract structural pattern was explicit or implicit to subjects. Note that the source waves arising from right primary/secondary visual cortices in response to standard and structural deviant stimuli did not differ at all, indicating that this extraction of the abstract structural pattern of Chinese characters was left-lateralized. Besides, no difference was observed between source waves originating from any level of the visual cortex to standard and deviant stimuli that violated abstract tonal patterns, until 768 ms when a late effect related to conscious detection of targets occurred at higher levels of the visual cortex. Note that at late stages (later than 698 ms after stimulus onset), responses arising from bilateral associative visual cortices to standard and target stimuli differed for both sessions, no matter the structural or tonal feature was attended to. Our findings support the primitive intelligence of visual cortex to rapidly extract abstract orthographic patterns of Chinese characters that might be engaged in further lexical processing. Our findings also suggest that this rapid extraction can take place implicitly during reading.