What are the most important and decisive parameters that determine the structure and the property of polymer nanocomposites (PNCs)? Previous studies answered that controlling the nanoparticle interface is critical, which can be achieved with a choice of a compatible nanoparticle, a proper surface modification, and a change in the polymer chain length. In addition to these parameters, the processing condition of PNCs has recently emerged as an influential parameter for controlling PNC properties, suggesting the existence of the nonequilibrium effect of PNCs. In this regard, we chose the solvent as a main change in the processing condition and investigated the initial solvent-driven nonequilibrium effect of PNCs with varied nanoparticle (NP) sizes. We found that the type of the initial solvent is indeed crucial in determining the ultimate properties of the PNCs, and this becomes more influential as the size of NPs decreases. The decreasing size of NPs causes a conformational change in the adsorbed polymers from tightly packed layers to loosely dangling chains. This results in much greater differences in NP microstructures and rheological properties of PNCs, indicating a stronger nonequilibrium effect with smaller NPs.