AbstractResponse surface method of experimental design was applied to optimize the mechanical properties of polypropylene (PP)/nanoclay/CaCO3 hybrid ternary nanocomposite using three different levels of melt flow index (MFI) of PP, nanoclay, and CaCO3 contents. The samples were prepared by melt mixing in a lab scale corotating twin screw extruder. The main effect of each parameter on the tensile modulus, tensile strength, and impact strength was extensively discussed. The structure of obtained nanocomposite was studied using X‐ray diffraction (XRD), atomic force microscopy (AFM), and scanning electron microscopy (SEM) techniques. Tensile modulus and impact resistance of prepared ternary nanocomposite were correlated to considered parameters using a second‐order polynomial model. Also, the optimum values of studied variables were determined using contour plots. The obtained results show that increasing the nanoclay and CaCO3 contents improve the tensile modulus up to 45%, whereas the optimum value of impact strength, about 54%, is achieved at low concentrations of nanoclay (2 wt %) and CaCO3 (8 wt %). © 2011 Wiley Periodicals, Inc. J Appl Polym Sci, 2011