Accurate quantitative modeling of point defect and impurity aggregation during silicon crystal growth and wafer annealing requires a detailed understanding of the underlying atomic scale mechanisms involved in defect formation, diffusion, and clustering. Examples are presented that demonstrate the utility of atomic scale studies for generating a complete picture of defect aggregation in crystalline silicon. In the first example, an approach for computing the thermodynamic properties of point defect clusters at high temperature is presented that accounts for cluster configurational entropy. In the second example, a lattice kinetic Monte Carlo model is applied to the direct simulation of vacancy clustering in the presence of oxygen atoms, which are known to act as reversible vacancy traps. The simulations are able to capture complex aggregate morphologies that have been observed experimentally, in particular the cloud-like distribution of small clusters around voids, and the double-void structures frequently observed in Czochralski crystal growth.