We consider the influence of chaotic classical dynamics on the spectrum and conducting properties of semiconductor quantum dots in the nearly-isolated (Coulomb blockade) regime. In this regime the amplitude of the Coulomb blockade oscillations are found to fluctuate strongly from peak to peak and as a function of magnetic field. Microscopic calculations are performed on Robnik's model, a class of billiards obtained by quadratic conformal transformations of the unit disk. It is known that small distortions of this system from cylindrical symmetry lead to fully chaotic classical dynamics and a quantum spectrum well-described by the Wigner-Dyson random matrix ensembles. We show that in the chaotic regime the levels are rapidly fluctuating as a function of shape and comment on the implications of these results for quantum dot superlattices. We derive a universal one-parameter distribution describing the fluctuations of the resonance amplitudes in the regime where a single thermally-broadened level dominates. Breaking time-reversal symmetry with a weak magnetic field is shown to change this distribution substantially, an effect which should be experimentally observable. We find excellent agreement between numerical results from Robnik's model and the random-matrix model in the chaotic limit, and the expected disagreement in the regular limit. This indicates that this fluctuation phenomenon can distinguish between regular and chaotic dot potentials.