Dose delivery uncertainty is a major concern in proton therapy, adversely affecting the treatment precision and outcome. Recently, a promising technique, proton-acoustic (PA) imaging, has been developed to provide real-time in vivo 3D dose verification. However, its dosimetry accuracy is limited due to the limited-angle view of the ultrasound transducer. In this study, we developed a deep learning-based method to address the limited-view issue in the PA reconstruction. A deep cascaded convolutional neural network (DC-CNN) was proposed to reconstruct 3D high-quality radiation-induced pressures using PA signals detected by a matrix array, and then derive precise 3D dosimetry from pressures for dose verification in proton therapy. To validate its performance, we collected 81 prostate cancer patients’ proton therapy treatment plans. Dose was calculated using the commercial software RayStation and was normalized to the maximum dose. The PA simulation was performed using the open-source k-wave package. A matrix ultrasound array with 64 × 64 sensors and 500 kHz central frequency was simulated near the perineum to acquire radiofrequency (RF) signals during dose delivery. For realistic acoustic simulations, tissue heterogeneity and attenuation were considered, and Gaussian white noise was added to the acquired RF signals. The proposed DC-CNN was trained on 204 samples from 69 patients and tested on 26 samples from 12 other patients. Predicted 3D pressures and dose maps were compared against the ground truth qualitatively and quantitatively using root-mean-squared-error (RMSE), gamma-index (GI), and dice coefficient of isodose lines. Results demonstrated that the proposed method considerably improved the limited-view PA image quality, reconstructing pressures with clear and accurate structures and deriving doses with a high agreement with the ground truth. Quantitatively, the pressure accuracy achieved an RMSE of 0.061, and the dose accuracy achieved an RMSE of 0.044, GI (3%/3 mm) of 93.71%, and 90%-isodose line dice of 0.922. The proposed method demonstrates the feasibility of achieving high-quality quantitative 3D dosimetry in PA imaging using a matrix array, which potentially enables the online 3D dose verification for prostate proton therapy.