The dark-field signal measures the small-angle scattering strength and provides complementary diagnostic information. This is of particular interest for lung imaging due to the pronounced small-angle scatter from the alveolar microstructure. However, most dark-field imaging techniques are relatively complex, dose-inefficient, and require sophisticated optics and highly coherent X-ray sources. Speckle-based imaging promises to overcome these limitations due to its simple and versatile setup, only requiring the addition of a random phase modulator to conventional X-ray equipment. We investigated quantitatively the influence of sample structure, setup geometry, and source energy on the dark-field signal in speckle-based X-ray imaging with wave-optics simulations for ensembles of micro-spheres. We show that the dark-field signal is accurately predicted via a model originally derived for grating interferometry when using the mean frequency of the speckle pattern power spectral density as the characteristic speckle size. The size directly reflects the correlation length of the diffuser surface and did not change with energy or propagation distance within the near-field. The dark-field signal had a distinct dependence on sample structure and setup geometry but was also affected by beam hardening-induced modifications of the visibility spectrum. This study quantitatively demonstrates the behavior of the dark-field signal in speckle-based X-ray imaging.