In this paper we show that tethering of heterochromatic regions to nuclear landmarks and random encounters of chromosomes in the confined nuclear volume are sufficient to explain the higher-order organization of the budding yeast genome. We have quantitatively characterized the contact patterns and nuclear territories that emerge when chromosomes are allowed to behave as constrained but otherwise randomly configured flexible polymer chains in the nucleus. Remarkably, this constrained random encounter model explains in a statistical manner the experimental hallmarks of the S. cerevisiae genome organization, including (1) the folding patterns of individual chromosomes; (2) the highly enriched interactions between specific chromatin regions and chromosomes; (3) the emergence, shape, and position of gene territories; (4) the mean distances between pairs of telomeres; and (5) even the co-location of functionally related gene loci, including early replication start sites and tRNA genes. Therefore, most aspects of the yeast genome organization can be explained without calling on biochemically mediated chromatin interactions. Such interactions may modulate the pre-existing propensity for co-localization but seem not to be the cause for the observed higher-order organization. The fact that geometrical constraints alone yield a highly organized genome structure, on which different functional elements are specifically distributed, has strong implications for the folding principles of the genome and the evolution of its function.