The main challenges encountered in the development of dental composites are to formulate the composites with minimum polymerization shrinkage without sacrificing degree of conversion. To overcome this problem, various fillers and composite materials have been proposed till date, but the analysis was limited. Hence, our main objective of this study was to investigate the polymerization shrinkage and mechanical behavior of resin-based dental composite by means of adding different weight fraction of silane-modified nanosilica particles. The dental composite consisting of a monomer system (50 wt.% bisphenol-A glycidyl methacrylate (Bis-GMA), 49 wt.% tri-ethylene glycol dimethacrylate (TEGDMA), 0.2 wt.-% Camphorquinone and 0.8 wt.-% Ethyl 4 dimethyl amino benzoate) was filled with different weight percentage of silane-modified nanosilica particles. Fourier transform infrared (FTIR) spectroscopy and transmission electron microscope (TEM) analysis of modified nanosilica particles clearly indicated the presence of C = O bond of carbonyl group and stable siloxane linkage between the monomer and filler particle. The mechanical results indicated that the composite with 3 wt.% nanosilica particle exhibited maximum depth of cure (4.2 mm) and the lowest polymerization shrinkage (1.53%). However, unfilled dental composite exhibited maximum compression strength (464 MPa) which was approximately 55% more than the compressive strength of filled composite. Finally, Dynamic mechanical analysis (DMA) results revealed that 1 wt.-% nanosilica content recorded the highest value of storage modulus. The proposed dental material exhibited better mechanical properties and lowest polymerization shrinkage with maximum conversion.