Objective We compared the performance on the Rey–Osterrieth Complex Figure Test (ROCF) of patients that had undergone unilateral anterior temporal lobectomy under both Taylor’s and Loring’s scoring systems to identify the sensitivity and specificity of each item for differentiating visuospatial memory deficits. Method We administered the ROCF to evaluate the visual memory of 37 left anterior temporal lobectomy (LATL) and 38 right anterior temporal lobectomy (RATL) patients with unilateral temporal lobe epilepsy who had undergone a standard unilateral anterior temporal lobectomy between 1996 and 2010. Fisher’s exact and Qui–Quadrado tests were used to analyze the relationships between the qualitative variables. The Mann–Whitney U test was used to compare the quantitative variables from the right and left sides. Results RATL patients performed worse than LATL patients based on the total score for delayed recall (DR) (p = 0.012). The scoring system’s showed a specificity of 97.2% & 78.9% and sensitivity of 10.5% & 62.2% on DR, for the Taylor and Loring systems respectively. Our detailed analysis of certain items showed that some differed between the groups in terms of the presence/absence, correct reproduction, and errors of those items. Loring’ errors I, IV, and X on DR and errors IV and X on immediate recall were more frequent in the RATL group. Conclusions The use of these two scoring systems combined may help maximize sensitivity and specificity with clinical populations. Further, our analyses showed that items could be clustered better and different weights could be given to them to maximize sensitivity and specificity.