Cancer patients face a four- to sevenfold higher risk of venous thromboembolism (VTE) than the general population. Novel oral anticoagulants (NOACs) provide convenient alternatives to traditional therapies. We performed a systematic literature search across PubMed, Embase, and the Cochrane Library, targeting studies that examined the use of NOACs in cancer-associated VTE. The search included randomized controlled trials (RCTs). Selected studies compared NOACs with low-molecular-weight heparin (LMWH) or vitamin K antagonists (VKA) in cancer patients diagnosed with VTE. A meta-analysis using a random-effects model was applied to estimate pooled effect sizes for outcomes. In this meta-analysis, we included 12 RCTs. Results showed NOACs were more effective than LMWH in preventing VTE recurrence (RR 0.66, 95% CI 0.52-0.83, p = 0.0004). Compared with VKAs, NOACs showed no significant difference (RR 0.63, 95% CI 0.34-1.15, p = 0.13). However, this finding is limited by the small patient sample. Major bleeding outcomes were similar between NOACs and LMWH/VKAs (RR 1.24, 95% CI 0.85-1.80, p = 0.28; RR 0.77, 95% CI 0.39-1.53, p = 0.46, respectively). Meta-regression analysis indicated a statistically significant positive correlation between mortality and major bleeding events when comparing NOACs with LMWH (p = 0.049). There was no significant difference in all-cause mortality between patients treated with NOACs and those treated with LMWH (RR 1.04, 95% CI 0.92-1.18, p = 0.54) or VKAs (RR 0.94, 95% CI 0.72-1.23, p = 0.65). Meta-analysis shows NOACs, especially factor Xa inhibitors, reduce VTE recurrence in cancer patients more effectively than LMWH. Comparison between NOACs and VKAs is inconclusive due to limited patient data. Further research is needed to assess NOACs' efficacy and safety against VKAs.