Self-incompatibility (SI) is a major issue in dragon fruit (Selenicereus spp.) breeding and production. Therefore, a better understanding of the dragon fruit SI mechanism is needed to improve breeding efficiency and ultimate production costs. To reveal the underlying mechanisms of SI in dragon fruit, plant anatomy, de novo RNA sequencing-based transcriptomic analysis, and multiple bioinformatic approaches were used to analyze gene expression in the pistils of the self-pollinated and cross-pollinated dragon fruit flowers at different intervals of time after pollination. Using fluorescence microscopy, we observed that the pollen of 'Hongshuijing', a self-incompatible dragon fruit variety (S. monacanthus), germinated on its own stigma. However, the pollen tube elongation has ceased at 1/2 of the style, confirming that dragon fruit experiences gametophyte self-incompatibility (GSI). We found that the pollen tube elongation in vitro was inhibited by self-style glycoproteins in the SI variety, indicating that glycoproteins were involved in SI. That is to say the female S factor should be homologous of S-RNase or PrsS (P. rhoeas stigma S factor), both of which are glycoproteins and are the female S factors of the two known GSI mechanism respectively. Bioinformatics analyses indicated that among the 43,954 assembled unigenes from pistil, there were six S-RNase genes, while 158 F-box genes were identified from a pollen transcriptomic dataset. There were no P. rhoeas type S genes discovered. Thus, the identified S-RNase and F-box represent the candidate female and male S genes, respectively. Analysis of differentially expressed genes (DEGs) between the self and cross-pollinated pistils at different time intervals led to the identification of 6,353 genes. We then used a weighted gene co-expression network analysis (WGCNA) to find some non-S locus genes in SI responses in dragon fruit. Additionally, 13 transcription factors (TFs) (YABBY4, ANL2, ERF43, ARF2, BLH7, KNAT6, PIF3, two OBF1, two HY5 and two LHY/CCA) were identified to be involved in dragon fruit GSI. Thus, we uncovered candidate S and non-S genes and predicted more SI-related genes for a more detailed investigation of the molecular mechanism of dragon fruit SI. Our findings suggest that dragon fruit possesses a GSI system and involves some unique regulators. This study lays the groundwork for future research into SI mechanisms in dragon fruit and other plant species.