Water released from subducting slabs through a dehydration reaction may lower the viscosity of the mantle significantly. Thus, we may expect a low viscosity wedge (LVW) above the subducting slabs. The LVW coupled with a large-scale flow induced by the subducting slabs may allow the existence of roll-like small-scale convection whose axis is normal to the strike of the plate boundary. Such a roll structure may explain the origin of along-arc variations of mantle temperature proposed recently in northeast Japan. We study this possibility using both 2D and 3D models with/without pressure- and temperature-dependent viscosity. 2D models without pressure and temperature dependence of viscosity show that, with a reasonable geometry of the LVW and subduction speed, small-scale convection is likely to occur when the viscosity of the LVW is less than 1019 Pa s. Corresponding 3D model studies reveal that the wavelength of rolls depends on the depth of the LVW. The inclusion of temperature-dependent viscosity requires the existence of further low viscosity in the LVW, since temperature dependence suppresses the instability of the cold thermal boundary layer. Pressure (i.e. depth) dependence coupled with temperature dependence of the viscosity promotes short wavelength instabilities. The model, which shows a relatively moderate viscosity decrease in the LVW (most of the LVW viscosity is 1018∼1019 Pa s) and a wavelength of roll ∼80 km, has a rather small activation energy and volume (∼130 kJ/mol and ∼4 cm3/mol) of the viscosity. This small activation energy and volume may be possible, if we regard them as an effective viscosity of non-linear rheology.