The anti-adjacency matrix (or eccentricity matrix) of a graph is obtained from its distance matrix by retaining for each row and each column only the largest distances. This matrix can be viewed as the opposite of the adjacency matrix, which is, on the contrary, obtained from the distance matrix of a graph by keeping for each row and each column only the distances being 1. In this paper, we prove that the graphs with exactly one positive anti-adjacency eigenvalue are determined by the anti-adjacency spectra. As corollaries, the well-known (generalized) friendship graphs and windmill graphs are shown to be determined by their anti-adjacency spectra.