Nanoparticles and amorphous materials are common constituents of the shallow sections of active faults. Understanding the conditions at which nanoparticles are produced and their effects on friction can further improve our understanding of fault mechanics and earthquake energy budgets. Here we present the results of 59 rotary shear experiments conducted at room humidity conditions on gouge consisting of mixtures of smectite (Ca-montmorillonite) and quartz. Experiments with 60, 50, 25, 0 wt.% Ca-montmorillonite, were performed to investigate the influence of variable clay content on nanoparticle production and their influence on frictional processes. All experiments were performed at a normal stress of 5 MPa, slip rate of 0.0003≤V≤1.5 ms−1, and at a displacement of 3 m. To monitor the development of fabric and the mineralogical changes during the experiments, we investigated the deformed gouges using scanning and transmission electron microscopy combined with X-ray powder diffraction quantitative phase analysis. This integrated analytical approach reveals that, at all slip rates and compositions, the nanoparticles (grain size of 10–50 nm) are partly amorphous and result from cataclasis, wear and mechanical solid-state amorphization of smectite. The maximum production of amorphous nanoparticle occurs in the intermediate slip rate range (0.0003≤V≤0.1 ms−1), at the highest frictional work, and is associated to diffuse deformation and slip strengthening behavior. Instead, the lowest production of amorphous nanoparticles occurs at co-seismic slip rates (V≥1.3 ms−1), at the highest frictional power and is associated with strain and heat localization and slip weakening behavior. Our findings suggest that, independently of the amount of smectite nanoparticles, they produce fault weakening only when typical co-seismic slip rates (>0.1 ms−1) are achieved. This implies that estimates of the fracture surface energy dissipated during earthquakes in natural faults might be extremely difficult to constrain.