Ultrasound backscattering signals from material microstructures can be used to evaluate the material microstructure grain size. This typically involves making pulse-echo immersion measurements at multiple locations using a focused ultrasonic transducer in order to obtain an accurate estimate of the root-mean-square amplitude of the back-scattered signal at a specified focal position. However, this restricts some practical applications of using such techniques in, for example, on-line measurements in high-value manufacturing and in-service inspections where multiple immersion measurements are not feasible to use. The main benefit of using ultrasonic phased arrays is that one array probe at one position can focus ultrasound beams at multiple points using different focal laws either physically or in data postprocessing. Potentially this means that accurate grain size measurements can be obtained from a single array measurement. In this paper, the classic backscattering method for conventional transducers is adapted to be used for full matrix capture datasets from an ultrasonic array. Three-dimensional ultrasonic models are developed in the proposed inverse process to measure material microstructure grain size. Experimental validations were performed on two metallic materials: copper (EN1652) and bright mild steel (BS970). A good agreement is shown between the experimentally measured grain sizes from array data and metallography measurements. Compared to the classic pulse-echo immersion back-scattering measurements, the proposed method enables accurate measurement of grain size in a direct contact configuration at fewer locations. This has potential to make on-line grain size measurements possible.