The current brachytherapy dose calculation is still based on the oversimplified water kernel superposition algorithm recommended by AAPM TG43. Monte Carlo (MC) method provides high accuracy in dose calculation. However, the MC simulation is computationally intensive and too slow to use in the time-sensitive clinical workflow. This study aims to develop a fast neural network (SK-UNet) to predict 3D brachytherapy dose distribution calculated by the MC simulations. We hypothesize that the tracks of Ir-192 inside applicators are essential in dose map prediction and the Selective Kernel Networks (SK) can be employed to emphasize the contribution of the applicator in training. We then included SK in UNet to select appropriate receptive field sizes as needed. The network inputs were CT images of patient anatomy, binary masks of HRCTV, bladder, rectum, and source tracks, and the output was the predicted dose map. Totally, 120 cases were used for training and 30 for testing. All clinically common applicators (e.g., vaginal, tandem and ovoid, multi-channel applicator, free needles, etc.) were involved in the study. Model performance was evaluated by the mean absolute error (MAE) of DVH dosimetric metrics and dose distribution metrics between ground truth (GT) and prediction. A smaller MAE indicated a more accurate dose prediction. Dose distribution metrics include the overdose volume index (ODI), target conformity (TC), dose homogeneity index (DHI), and conformal index (COIN). GT was dose calculated by MC simulations. 3D Gamma analysis was also involved. As shown in Table 1, compared with GT, SK-Net showed comparable DVH dosimetric metrics with 0.28±0.19 deviation for HRCTV D90%, 0.21±0.16 deviation for bladder D2cc, and 0.25±0.22 deviation for rectum D2cc. In dose distribution metrics, the deviations between GT and prediction were TC = 0.04±0.02, ODI = 0.03±0.03, DHI = 0.02±0.02, and COIN = 0.02±0.03. The gamma passing rate was 93%±6%. The 3D dose map prediction for each patient takes about 6s on average in an NVIDIA GeForce RTX 3060 GPU compared to hours with MC simulations. SK-Net demonstrated comparable performance to the MC simulation but with a significantly shorter execution time, taking only 6 seconds on average. It offered a solution to the trade-off between accuracy and speed and has the potential to serve as an alternative to the time-consuming MC simulation in brachytherapy. The developed technique is believed to have the potential for future application with other types of radiation sources and cancers.