Top-down processing in neocortex underlies functions such as prediction, expectation, and attention. Visual systems have much feedback connection that carries information of behavioral context. Top-down signals along feedback pathways modulate the representation of visual information in early visual areas such as primary visual cortex (V1). Recent studies have shown further that beta rhythms are responsible for the transmission of behavioral-context information to lower visual areas. However, the mechanism underlying top-down influence and the role of brain rhythms in top-down processing are poorly understood. To address these issues, we focus on experimental studies on top-down influence in visual perceptual tasks. We develop a model of visual system, in which early visual areas are subjected to top-down influence from a recognition area. We show that task-relevant information in early visual areas is regulated by a push-pull effect, produced by somatostatin-expressing interneurons and top-down signal. We also show that task-context information is coordinated by the phase-phase coupling of beta rhythms, while the local, task-relevant stimulus features are enhanced by the phase-amplitude coupling of beta and gamma rhythms. Furthermore, the feedback from a higher visual area such as secondary visual area facilitates the gating of task-relevant information in V1. The results provide insights to understanding the roles of inhibitory interneurons and brain rhythms in top-down influence on information processing in early visual areas.