Indigenous bacteriophage communities (virome) in the human gut have a huge impact on the structure and function of gut bacterial communities (bacteriome), but virome variation at a population scale is not fully investigated yet. Here, we analyse the gut dsDNA virome in the Japanese 4D cohort of 4198 deeply phenotyped individuals. By assembling metagenomic reads, we discover thousands of high-quality phage genomes including previously uncharacterised phage clades with different bacterial hosts than known major ones. The distribution of host bacteria is a strong determinant for the distribution of phages in the gut, and virome diversity is highly correlated with anti-viral defence mechanisms of the bacteriome, such as CRISPR-Cas and restriction-modification systems. We identify 97 various intrinsic/extrinsic factors that significantly affect the virome structure, including age, sex, lifestyle, and diet, most of which showed consistent associations with both phages and their predicted bacterial hosts. Among the metadata categories, disease and medication have the strongest effects on the virome structure. Overall, these results present a basis to understand the symbiotic communities of bacteria and their viruses in the human gut, which will facilitate the medical and industrial applications of indigenous viruses.