A pivotal nanofiller network will be constructed by the filler loading threshold inside the silicone rubber, leading to abrupt enhancement in the rheological properties of the composites. However, the contribution of the nanofiller network to the performance mutation is poorly understood due to lack of direct evidence to recognize the formation of filler networks. This work quantitatively investigated the filler aggregation network of solvent-extracted monodisperse silica-filled polydimethylsiloxane (PDMS) composites to interpret the rheological properties. The results indicated that, when filler loadings reach 60 phr, the size of the filler network reaches its maximum (1280.5 nm), significantly increasing the storage modulus (166 kPa) and Payne effect (163 kPa), due to the formation of a filler network confirmed by Dynamic Light Scattering (DLS) and scanning electron microscope (SEM) observation. The reduction in aggregate size observed with longer extraction times is because of the collapse of the nanofiller network, which occurs as the polymer chains are removed. The aggregates reappear in a monodisperse form as the extraction duration reaches 20 days. This confirms that filler aggregates of interconnected polymer chains can form a well-developed network structure that effectively supports and transfers stresses. This contributes to an in-depth understanding of the formation mechanism of nanofiller networks, aiding the advancement of high-performance polymer nanocomposites.