We present a source catalogue and first results from a deep, blind radio survey carried out at 20 GHz with the Australia Telescope Compact Array, with follow-up observations at 5.5, 9 and 18 GHz. The Australia Telescope 20 GHz (AT20G) deep pilot survey covers a total area of 5 deg2 in the Chandra Deep Field South and in Stripe 82 of the Sloan Digital Sky Survey. We estimate the survey to be 90 per cent complete above 2.5 mJy. Of the 85 sources detected, 55 per cent have steep spectra (|$\alpha _{1.4}^{20} < -0.5$|) and 45 per cent have flat or inverted spectra (|$\alpha _{1.4}^{20} \ge -0.5$|). The steep-spectrum sources tend to have single power-law spectra between 1.4 and 18 GHz, while the spectral indices of the flat- or inverted-spectrum sources tend to steepen with frequency. Among the 18 inverted-spectrum (|$\alpha _{1.4}^{20} \ge 0.0$|) sources, 10 have clearly defined peaks in their spectra with |$\alpha _{1.4}^{5.5} > 0.15$| and |$\alpha _{9}^{18} < -0.15$|. On a 3-yr time-scale, at least 10 sources varied by more than 15 per cent at 20 GHz, showing that variability is still common at the low flux densities probed by the AT20G-deep pilot survey. We find a strong and puzzling shift in the typical spectral index of the 15–20-GHz source population when combining data from the AT20G, Ninth Cambridge and Tenth Cambridge surveys: there is a shift towards a steeper-spectrum population when going from ∼1 Jy to ∼5 mJy, which is followed by a shift back towards a flatter-spectrum population below ∼5 mJy. The 5-GHz source-count model by Jackson & Wall, which only includes contributions from FRI and FRII sources, and star-forming galaxies, does not reproduce the observed flattening of the flat-spectrum counts below ∼5 mJy. It is therefore possible that another population of sources is contributing to this effect.