The ability to group items and events into functional categories is a fundamental function for visual recognition. Experimental studies have shown the different roles in information representations of inferior temporal (IT) and prefrontal cortices (PFC) in a categorization task. However, it remains elusive how category information is generated in PFC and maintained in a delay period and how the interaction between IT and PFC influences category performance. To address these issues, we develop a network model of visual system, which performs a delayed match-to-category task. The model consists of networks of V4, IT, and PFC. We show that in IT visual information required for categorization is represented by a combination of prototype features. We also show that category information in PFC is represented by two dynamical attractors weakly linked, resulting from the difference in firing thresholds of PFC neurons. Lower and higher firing thresholds contribute to working memory maintenance and decision-making, respectively. Furthermore, we show that top-down signal from PFC to IT improves the ability of PFC neurons to categorize the mixed images that are closer to a category boundary. Our model may provide a clue for understanding the neural mechanism underlying categorization task.