Ultramylonites are among the most extreme fault rocks that commonly occur in the mid-crustal brittle–plastic transition and are mainly characterized by intensely sheared fine-grained microstructures and well-mixed mineral phases. Although the deformation mechanism of ultramylonites is key to understanding the rheological behavior of the mid-crustal shear zone, their microstructural development is still controversial owing to their intensely fine-grained textures. To investigate the possible crustal deformation mechanisms, we studied 13 mylonites obtained from the Kashio shear zone along the Median Tectonic Line that is the largest strike-slip fault in Japan. In particular, we investigated various mixed quartz–plagioclase layers developed within tonalitic mylonite, which are representative of the common mean grain size and crystal fabric of quartz among the studied samples. A high-quality phase-orientation map obtained by electron backscattered diffraction showed not only a wide range of quartz–plagioclase mixing (10%–80% in quartz modal composition) but also revealed a correlation between grain size reduction and crystal fabric weakening in quartz, indicating a change in the deformation mechanism from dislocation creep to grain-size-sensitive creep in the mixed quartz-plagioclase layers. In contrast, plagioclase showed an almost consistent fine grain size and weak to random crystal fabrics regardless of modal composition, indicating that grain size-sensitive creep is dominant. Combined with laboratory-determined flow laws, our results show that the Kashio shear zone could have developed under deformation mechanisms in which the viscosities of quartz and plagioclase are nearly comparable, effectively within 1017–1019 Pa·s, thereby possibly enabling extensive shearing along the Median Tectonic Line.