Mutations in the β-subunit of RNA polymerase (RNAP), encoded by rpoB, are responsible for rifampicin resistance (Rif(R)). Although many mutations in rpoB can reduce susceptibility, only a few are frequent amongst Rif(R) clinical Mycobacterium tuberculosis (MTB) isolates. It has been suggested that there is a negative correlation between the fitness costs of Rif(R) mutations and their respective clinical frequency, but so far comparable fitness cost measurements have only been conducted for a very limited number of Rif(R) mutations. We tested this hypothesis using Salmonella and Mycobacterium smegmatis as model organisms. We constructed 122 different Rif(R) mutations in Salmonella. MICs and relative fitness costs in the presence and absence of rifampicin were determined for each mutant, including for a smaller number of Rif(R) M. smegmatis strains. Results were compared with available mutation frequency data from clinical MTB isolates. (i) Rif(R) mutations frequently found in MTB isolates have a fitness cost in Salmonella Typhimurium and M. smegmatis. (ii) Clinically frequent Rif(R) mutations have a high rifampicin MIC. (iii) There is a strong correlation between the magnitude of the fitness cost of a Rif(R) mutation in Salmonella Typhimurium or M. smegmatis and the frequency with which that mutation is associated with secondary (putative compensatory) mutations in RNAP of clinical MTB isolates. This suggests that the success of Rif(R) mutations in clinical MTB isolates may be dependent not only on a low initial fitness cost, but rather the results of three factors: (i) a high rifampicin MIC; (ii) a relatively low initial fitness cost; and (iii) the ability to additionally acquire compensatory mutations selected to further reduce fitness cost.