Seed implant brachytherapy (SIBT) is a promising treatment modality for parotid gland cancers (PGCs). However, the current clinical standard dose calculation method based on the American Association of Physicists in Medicine (AAPM) Task Group 43 (TG-43) Report oversimplifies patient anatomy as a homogeneous water phantom medium, leading to significant dose calculation errors due to heterogeneity surrounding the parotid gland. Monte Carlo Simulation (MCS) can yield accurate dose distributions but the long computation time hinders its wide application in clinical practice. This paper aims to develop an end-to-end deep convolutional neural network-based dose engine (DCNN-DE) to achieve fast and accurate dose calculation for PGC SIBT. A DCNN model was trained using the patient's CT images and TG-43-based dose maps as inputs, with the corresponding MCS-based dose maps as the ground truth. The DCNN model was enhanced based on our previously proposed model by incorporating attention gates (AGs) and large kernel convolutions. Training and evaluation of the model were performed using a dataset comprising 188 PGC I-125 SIBT patient cases, and its transferability was tested on an additional 16 non-PGC head and neck cancers (HNCs) I-125 SIBT patient cases. Comparison studies were conducted to validate the superiority of the enhanced model over the original one and compare their overall performance. On the PGC testing dataset, the DCNN-DE demonstrated the ability to generate accurate dose maps, with percentage absolute errors (PAEs) of 0.67%±0.47% for clinical target volume (CTV) D90 and 1.04%±1.33% for skin D0.1cc. The comparison studies revealed that incorporating AGs and large kernel convolutions resulted in 8.2% (p<0.001) and 3.1% (p<0.001) accuracy improvement, respectively, as measured by dose mean absolute error. On the non-PGC HNC dataset, the DCNN-DE exhibited good transferability, achieving a CTV D90 PAE of 1.88%±1.73%. The DCNN-DE can generate a dose map in less than 10 ms. We have developed and validated an end-to-end DCNN-DE for PGC SIBT. The proposed DCNN-DE enables fast and accurate dose calculation, making it suitable for application in the plan optimization and evaluation process of PGC SIBT.