<h3>Purpose/Objective(s)</h3> This study aims to predict voxel-wise therapy response PET image for anal cancer radiotherapy utilizing pre-radiotherapy PET/CT, planning targets and dose, toward estimating the treatment outcome and reducing hematologic toxicity. <h3>Materials/Methods</h3> 18FDG-PET/CT scans were obtained from locally advanced anal cancer patients with 2-week prior to (pre-RT), 12-week following treatment (post-RT). All PET/CT images were registered to the planning CT. Patients were treated with the IMRT with the mean dose of 55.8Gy. The prediction of post-RT PET image was constructed by 3D U-Net deep learning with five spatial levels. The input of 3D U-Net has seven separate channels, including PTV, V40, pelvic bone, bladder, Pre-RT PET, planning CT, and planned dose. All images were resized to the volumes with 128 × 128 × 128 voxels. The prediction was calculated in the volume of interested (VOI) where the dose is larger than 5% prescribed dose. To reduce the bias due to high bladder uptake, the SUV in the bladder of post-RT PET was reduced to the mean liver uptakes, and the resultant post-RT PET images were used in the study. Online data augmentation was applied during both training and testing stages. The performance of the prediction model was evaluated by the voxel-based mean absolute error and joint histograms. The ANVOA was used to identify difference between the predicted post-RT PET images and ground truth images for VOIs, including PTV and OARs. <h3>Results</h3> Fifty-five patients with pre-RT and post-RT PET images were included in the study. Among them, 30 patients were randomly selected as training data set, and 10 patients were used for the validation set, and 15 patients were used as the test set. For the evaluated volumes (> 5% prescribed dose), the similarity of predicted image and ground truth were evaluated by the joint histogram with voxelwise correlation (mean correlation coefficient, 91. 21% +/- 5.36). The difference between the mean SUV of predicated PET image and the ground truth was evaluated by ANOVA with p=0.422. In the volume of PTV, the mean SUV of predicted image was 1.10+/-0.11, and 1.08+/-0.30 for ground truth, p=0.988. In the volume of V40, the mean SUV of predicted image was 1.00+/-0.06, and 1.05+/-0.20 for ground truth, p=0.549. We also evaluated the mean uptake of pelvic bone marrow with results of 0.89+/-0.20 for predicated image, and 0.81+/-0.11 for ground truth, p=0.261. Our results show that there is no significant difference in the target volumes and OARs between predicted PET image and ground truth images. <h3>Conclusion</h3> Our approach can effectively predict the post-RT PET image based on the pre-treatment image and treatment information. 3D deep learning prediction of metabolic treatment outcome provides a feasible approach for evaluating efficacy of local control of anal cancer treatment. The results can be used to predict local failure of disease, toward implementing response-driven adaptive radiotherapy.