Collective motion provides a spectacular example of self-organization in Nature. Visual information plays a crucial role among various types of information in determining interactions. Recently, experiments have revealed that organisms such as fish and insects selectively utilize a portion, rather than the entirety, of visual information. Here, focusing on fish, we propose an agent-based model where the direction of attention is guided by visual stimuli received from the images of nearby fish. Our model reproduces a branching phenomenon where a fish selectively follows a specific individual as the distance between two or three nearby fish increases. Furthermore, our model replicates various patterns of collective motion in a group of agents, such as vortex, polarized school, swarm, and turning. We also discuss the topological nature of the visual interaction, as well as the positional distribution of nearby fish and the map of pairwise and three-body interactions induced by them. Through a comprehensive comparison with existing experimental results, we clarify the roles of visual interactions and issues to be resolved by other forms of interactions.