Purple acid phosphatases (PAPs) are ubiquitous binuclear metallohydrolases that have been isolated from various animals, plants and some types of fungi. In humans and mice, elevated PAP activity in osteoclasts is associated with osteoporosis, making human PAP an attractive target for the development of anti-osteoporotic drugs. Based on previous studies focusing on phosphonate scaffolds, as well as a new crystal structure of a PAP in complex with a derivative of a previously synthesized α-aminonaphthylmethylphosphonic acid, phosphonates 24–40 were designed as new PAP inhibitor candidates. Subsequent docking studies predicted that all of these compounds are likely to interact strongly with the active site of human PAP and most are likely to interact strongly with the active site of pig PAP. The seventeen candidates were synthesized with good yields and nine of them (26–28, 30, 33–36 and 38) inhibit in the sub-micromolar to nanomolar range against pig PAP, with 28 and 35 being the most potent mammalian PAP inhibitors reported with Ki values of 168 nM and 186 nM, respectively. This study thus paves the way for the next stage of drug development for phosphonate inhibitors of PAPs as anti-osteoporotic agents.