Colored rice is a type of high-quality, high-added-value rice that has attracted increasing attention in recent years. The use of large amounts of inorganic nitrogen fertilizer in rice fields results in low fertilizer use efficiency and high environmental pollution. Organic fertilizer is a promising way to improve soil quality and sustain high yields. However, most studies focus on the effect of animal-based organic fertilizers. The effects of different ratios of plant-based organic fertilizer and inorganic fertilizer on the grain yield and quality of colored rice have rarely been reported. Therefore, a two-year field experiment was conducted in 2020 and 2021 to study the effects of replacing inorganic N fertilizers with plant-based organic fertilizers on the yield, nitrogen use efficiency (NUE), and anthocyanin content of two colored rice varieties in a tropical region in China. The experimental treatments included no nitrogen fertilization (T1), 100% inorganic nitrogen fertilizer (T2), 30% inorganic nitrogen fertilizer substitution with plant-based organic fertilizer (T3), 60% inorganic nitrogen fertilizer substitution with plant-based organic fertilizer (T4), and 100% plant-based organic fertilizer (T5). The total nitrogen provided to all the treatments except T1 was the same at 120 kg ha–1. Our results showed that the T3 treatment enhanced the grain yield and anthocyanin content of colored rice by increasing nitrogen use efficiency compared with T2. On average, grain yields were increased by 9 and 8%, while the anthocyanin content increased by 16 and 10% in the two colored rice varieties under T3 across the two years, respectively, as compared with T2. Further study of the residual effect of partial substitution of inorganic fertilizers showed that the substitution of inorganic fertilizer with plant-based organic fertilizer improved the soil physio-chemical properties, and thus increased the rice grain yield, in the subsequent seasons. The highest grain yield of the subsequent rice crop was observed under the T5 treatment. Our results suggested that the application of plant-based organic fertilizers can sustain the production of colored rice with high anthocyanin content in tropical regions, which is beneficial in reconciling the relationship between rice production and environmental protection.