Semen production traits are important aspects of bull fertility, because semen quantity leads to direct profits for artificial insemination centres, and semen quality is associated with the probability of achieving a pregnancy. Most genome-wide association studies (GWASs) for semen production traits have assumed that each quantitative trait locus (QTL) has an additive effect. However, GWASs that account for non-additive effects are also important in fitness traits, such as bull fertility. Here, we performed a GWAS using models that accounted for additive and non-additive effects to evaluate the importance of non-additive effects on five semen production traits in beef and dairy bulls. A total of 65 463 records for 615 Japanese Black bulls (JB) and 50 734 records for 873 Holstein bulls (HOL), which were previously genotyped using the Illumina BovineSNP50 BeadChip, were used to estimate genetic parameters and perform GWAS. The heritability estimates were low (ranged from 0.11 to 0.23), and the repeatability estimates were low to moderate (ranged from 0.28 to 0.45) in both breeds. The estimated repeatability was approximately twice as high as the estimated heritability for all traits. In this study, only one significant region with an additive effect was detected in each breed, but multiple significant regions with non-additive effects were detected for each breed. In particular, the region at approximately 64 Mbp on Bos taurus autosome 17 had the highest significant non-additive effect on four semen production traits in HOL. The rs41843851 single nucleotide polymorphism (SNP) in the region had a much lower P-value for the non-additive effect (P-value = 1.1 × 10−31) than for the additive effect (P-value = 1.1 × 10−8) in sperm motility. The AA and AB genotypes on the SNP had a higher phenotype than the BB genotype in HOL, and there was no bull with the BB genotype in JB. Our results showed that non-additive QTLs affect semen production traits, and a novel QTL accounting for non-additive effects could be detected by GWAS. This study provides new insights into non-additive QTLs that affect fitness traits, such as semen production traits in beef and dairy bulls.