Inbreeding depression refers to the reduced performance arising from increased homozygosity, a phenomenon that is the reverse of heterosis and exists among plants and animals. As a natural self-pollinated crop with strong heterosis, the mechanism of inbreeding depression in rice is largely unknown. To understand the genetic basis of inbreeding depression, we constructed a successive inbreeding population from the F2 to F4 generation and observed inbreeding depression of all heterotic traits in the progeny along with the decay of heterozygosity in each generation. The expected depression effect was largely explained by 13 QTLs showing dominant effects for spikelets per panicle, 11 for primary branches, and 12 for secondary branches, and these loci constitute the main correlation between heterosis and inbreeding depression. However, the genetic basis of inbreeding depression is also distinct from that of heterosis, such that a biased transmission ratio of alleles for QTLs with either dominant or additive effects in four segregation distortion regions would result in minor effects in expected depression. Noticeably, two-locus interactions may change the extent and direction of the depression effects of the target loci, and overall interactions would promote inbreeding depression among generations. Using an F2:3 variation population, the actual performance of the loci showing expected depression was evaluated considering the heterozygosity decay in the background after inbreeding. We found inconsistent or various degrees of background depression from the F2 to F3 generation assuming different genotypes of the target locus, which may affect the actual depression effect of the locus due to epistasis. The results suggest that the genetic architecture of inbreeding depression and heterosis is closely linked but also differs in their intrinsic mechanisms, which expand our understanding of the whole-genome architecture of inbreeding depression.