Self-excited longitudinal combustion instabilities were investigated in a hypergolic liquid bipropellant combustor, which applied single dual-swirl coaxial injector. Hot-fire tests were conducted for four different injector geometries, while extensive tests on injection conditions were carried out for each injector geometry. The synchronous measurement of the pressure and heat release rate was applied, successfully capturing the process of the pressure and heat release rate enhanced coupling and developing into in-phase oscillation. By calculating Rayleigh index at the head and middle section of the chamber, it is shown that Rayleigh index of the middle section is even higher than that of the head, indicating a long heat release zone. When the combustion instability occurs, the pressure in propellant manifolds also oscillates with the same frequency and lags behind the oscillation in the combustor. Compared to the oscillation in the outer injector manifold, the oscillation in the inner injector manifold shows a higher correlation with that in the chamber in amplitude and phase. Based on numerical simulations of the multiphase cold flow inside the injector and combustion process in the chamber, it is found that injector geometries affect longitudinal combustion instability by changing spray cone angle. The spray with small cone angle is more sensitive to the modulation of longitudinal pressure wave in combustion simulations, which is more likely to excite the longitudinal combustion instability. Meanwhile, the combustion instability may be related to the pulsating coherent structure generated by the spray fluctuation, which is determined by injection conditions. Besides, a positive feedback closed-loop system associated with the active fluctuation and passive oscillation of the spray is believed to excite and sustain the longitudinal combustion instability.