With the objective of finding new classes of cognitive enhancers with potential for the treatment of neurodegenerative disorders, such as Alzheimer's disease, small molecule inhibitors of insulin-regulated aminopeptidase (IRAP) were designed and synthesized. IRAP is a member of the M1 family of zinc aminopeptidases and is abundantly expressed in areas of the brain associated with cognition, such as the amygdala, hippocampus and cerebral cortex. IRAP inhibitors were previously shown to enhance memory and learning in animal models. A comprehensive high throughput screening of 400,000 small molecules from the European Lead Factory library provided a series of 50 promising compounds in a qualified hit list (QHL). More than 30 IRAP inhibitors with an IC50 below 3.5 μM were identified. Herein, selected compounds from this QHL were assayed for solubility and permeability. Most of the selected compounds displayed good solubility, but further permeability studies on the best compounds revealed low blood brain barrier (BBB) permeability and high efflux in cells overexpressing P-gp pumps, rendering them less promising as starting points in drug discovery processes. Two compounds from the QHL were prioritized for further structural optimization; the pyridyl-substituted isoxazole 1a (QHL27) and the benzylhydroxamic acid derivative 1b (QHL1), both demonstrating fair BBB permeability and no indication of efflux. While our attempts to improve the isoxazole derivative 1a were not fruitful, a structural modification of 1b to provide the chloro-substituted benzylhydroxamic acid 14b resulted in a ten-fold improvement of the IRAP inhibition with an IC50 value of 60 nM. The binding modes of 1b and 14b were determined by free energy perturbation (FEP) analysis performed on candidate docking poses, determining a binding mode that accurately explained the experimental SAR. Further FEP studies of compound 14b suggested that it exhibits selectivity towards IRAP over Aminopeptidase N (APN), indicating its potential for targeted therapeutic applications.