O-GlcNAcylation is the dynamic and ubiquitous post-translational glycosylation of nucleocytoplasmic proteins on serine/threonine residues; it is implicated in regulation of the cell cycle. This protein modification is mainly governed by a pair of enzymes: O-GlcNAc transferase (OGT) adds the N-acetylglucosamine moiety to acceptor proteins, and O-GlcNAcase (OGA) hydrolyses the sugar moiety from protein acceptors. Irregular O-GlcNAcylation is linked to several diseases including cancer, diabetes and neurodegeneration. Recently, the discovery of small-molecule OGA inhibitors has enabled the physiological function of O-GlcNAcylation to be investigated. However, the design of highly potent and selective inhibitors faces several challenges as no full structural data of human OGA has been discovered to date. Moreover, there are a number of mechanistically similar related enzymes such as β-hexosaminidases (Hex), and the concomitant inhibition of these enzymes leads to undesirable lysosomal-storage disorders. This review highlights recent insights into the structure of human O-GlcNAcase and its isoforms. We focus on the catalytic mechanism and substrate recognition by OGA. In addition, it presents an updated overview of small-molecule OGA inhibitors, with either carbohydrate or noncarbohydrate scaffolds. We discuss inhibitor structures, binding modes, and selectivity towards the enzyme, and potential outcomes in probing O-GlcNAcylation at cellular levels.