Noninvasive brain diagnosis is extremely important because of its efficiency, low cost, and painless nature in the prediction of stroke, cerebral hemorrhage, and other brain research. At present, achieving full 3-D quantitative ultrasonic imaging of the human brain is a cutting-edge challenge due to the complex structures of the human brain and the strong scattering caused by the skulls. In this article, we achieved quantitative ultrasonic imaging of inside-brain anomalies with our proposed method, the decomposition descent learning-based full waveform inversion (DDL-FWI). The proposed method adopts a linear residual decomposing technique to greatly alleviate the computation burden in fast inversion tomography (FIT) with enhanced convergence guaranteed by residual functions. Testing results in both simulation and laboratory experiments demonstrated that our method can achieve high-quality quantitative imaging of brain soft tissues and skulls even starting from homogeneous water background in 2-D, and this method is capable of reconstructing both complex brain tissues and clots in 2-D and 3-D cases using either clean or noisy signals, with a robust 3-D clot resolution as small as 18 mm and 2-D reconstruction speed in 11.20 s. Combined with advanced ultrasonic hardware, DDL-FWI can be easily trained and used for brain imaging efficiently that frees patients from harmful influences from traditional imaging techniques, e.g., ionization radiations from X-ray computed tomography (CT).