We use measurements from the South Pole Telescope (SPT) Sunyaev–Zel'dovich (SZ) cluster survey in combination with X-ray measurements to constrain cosmological parameters. We present a statistical method that fits for the scaling relations of the SZ and X-ray cluster observables with mass while jointly fitting for cosmology. The method is generalizable to multiple cluster observables, and self-consistently accounts for the effects of the cluster selection and uncertainties in cluster mass calibration on the derived cosmological constraints. We apply this method to a data set consisting of an SZ-selected catalog of 18 galaxy clusters at z > 0.3 from the first 178 deg2 of the 2500 deg2 SPT-SZ survey, with 14 clusters having X-ray observations from either Chandra or XMM-Newton. Assuming a spatially flat ΛCDM cosmological model, we find the SPT cluster sample constrains σ8(Ωm/0.25)0.30 = 0.785 ± 0.037. In combination with measurements of the cosmic microwave background (CMB) power spectrum from the SPT and the seven-year Wilkinson Microwave Anisotropy Probe data, the SPT cluster sample constrains σ8 = 0.795 ± 0.016 and Ωm = 0.255 ± 0.016, a factor of 1.5 improvement on each parameter over the CMB data alone. We consider several extensions beyond the ΛCDM model by including the following as free parameters: the dark energy equation of state (w), the sum of the neutrino masses (Σmν), the effective number of relativistic species (Neff), and a primordial non-Gaussianity (fNL). We find that adding the SPT cluster data significantly improves the constraints on w and Σmν beyond those found when using measurements of the CMB, supernovae, baryon acoustic oscillations, and the Hubble constant. Considering each extension independently, we best constrain w = −0.973 ± 0.063 and the sum of neutrino masses Σmν < 0.28 eV at 95% confidence, a factor of 1.25 and 1.4 improvement, respectively, over the constraints without clusters. Assuming a ΛCDM model with a free Neff and Σmν, we measure Neff = 3.91 ± 0.42 and constrain Σmν < 0.63 eV at 95% confidence. We also use the SPT cluster sample to constrain fNL = −220 ± 317, consistent with zero primordial non-Gaussianity. Finally, we discuss the current systematic limitations due to the cluster mass calibration, and future improvements for the recently completed 2500 deg2 SPT-SZ survey. The survey has detected ∼500 clusters with a median redshift of ∼0.5 and a median mass of ∼2.3 × 1014 M☉ h−1 and, when combined with an improved cluster mass calibration and existing external cosmological data sets will significantly improve constraints on w.