Abstract As part of its ‘pro-innovation’ approach to artificial intelligence (AI), the UK has left public sector AI procurement and deployment to ‘regulation by contract’ based on thin guidance. Borrowing from the description of AI ‘hallucinations’ as plausible but incorrect answers given with high confidence by AI systems, I argue that this is a ‘regulatory hallucination’: an incorrect answer to the challenge of regulating the procurement and use of AI by the public sector. The pretence that public buyers can ‘confidently and responsibly procure AI technologies’ can generate individual harms and broader negative social effects as the public sector ramps up AI adoption and accumulates a potentially significant stock of AI deployments across all areas of public sector activity. I sketch an alternative strategy to boost the effectiveness of the goals of AI regulation and the protection of individual rights and collective interests through the creation of an independent authority.