The synergistic effects of Sm3+/Yb3+ on a p-n ternary heterostructure of Co3O4-In2O3 (Co-In) nanorods, prepared by a hydrothermal approach for the construction of a gas sensor with unique sensing characteristics are reported. The intrinsic characteristics of the ternary heterostructures, such as surface adsorption states, crystal structures, and chemical states, were analyzed. The sensing analyses demonstrated that amongst the Sm3+/Yb3+ incorporated in the Co-In surface, the Co-In 0.25 mol.% Sm3+ displayed superior gas sensing toward ethylbenzene gas at a low operational temperature (i.e., 75 °C). At 100 °C, the precise selectivity toward xylene vapour was witnessed for the Co-In: 0.25 mol.% YbO. At lower gas concentration ranges of 0.08–5 ppm of acetone, 0.5–5 ppm of benzene, and 5–10 ppm of toluene, ethylbenzene, and xylene, the Co-In: 0.25 mol.% YbO was more sensitive to 0.08 ppm of acetone at 75 °C. Collective materials characterization and testing of the device analyses demonstrated that the observed sensing enhancement is primarily ascribed to the stronger capacity of oxygen-adsorption on Co-In: 0.25 mol.%:YbO. The depletion layer width strongly increased due to the formation of the multi-ternary junctions between the Co3O4, In2O3, and Yb2O3. The VO-rich Co3O4 has favoured the adsorption of xylene. The lower detection of acetone at ppb was associated with the preferred facet of acetone detection in comparison to xylene. The prospective sensing mechanism of the ternary heterostructure-based sensors has also been proposed.