A computational study of three-dimensional instability of steady flows in a helical pipe of arbitrary curvature and torsion is carried out for the first time. The problem is formulated in Germano coordinates in two equivalent but different forms of the momentum equations so that results obtained using both formulations cross verify each other. An additional formulation in the cylindrical coordinates is applied for a limiting case of the toroidal pipe. The calculations are performed by the finite volume and finite difference methods. Grid independence of the results is established for both steady flows, the eigenvalues associated with the linear stability problem, and the critical parameters. The calculated steady flows agree well with experimental measurements and previous numerical results. The computed critical Reynolds numbers corresponding to the onset of oscillatory instability agree well with the most recent experimental results, but disagree with the earlier ones. Novel results related to the parametric stability study are reported.