We present the linear theory of the starting current of Cherenkov-cyclotron and Cherenkov instabilities generated by an electron beam passing through a metamaterial-loaded waveguide. Effective medium theory is used to represent the metamaterial structure properties. The theory predicts that the instabilities compete with the Cherenkov-cyclotron mode dominating at a lower magnetic field and the Cherenkov instability at a higher magnetic field. The theoretical results are compared to results from recent experiments at MIT using a 490 kV, 84 A electron beam in magnetic fields of 300 G to 1500 G. For an effective medium model fitted to the MIT experimental parameters, theory predicts that the Cherenkov-cyclotron mode will dominate below 780 G and the Cherenkov mode above 780 G, in good agreement with experimental observations of switching between these modes at 750 G. The analytical theory allows a better understanding of the mode competition and the dependence of the instabilities on key parameters such as voltage, current, and magnetic fields.