IMap and IMap2 Produce Erroneous Statistical Maps of Eye-Movement Differences

Abstract

The programs iMAP and iMAP2, developed by Caldara and Miellet (2011 Behavior Research Methods 43 846-878), have attempted to implement a general approach to the analysis of eye-movement data, providing not only 'heat maps' of areas of greater and lesser activity but also, and potentially of great practical importance, significance tests which take into account spatial autocorrelation in fixation locations. The tests in particular allow different groups to be compared, as in one of Caldara and Miellet's example datasets where fixation patterns of Western Caucasian and East Asian participants are said to be significantly different. The present paper argues that the significance tests, as implemented, used an inappropriate algorithm and therefore gave erroneous results. In particular, if participants are randomly allocated to two groups, which is a conventional randomization test, then in every case the program claimed to find 'significant differences', which cannot be correct. A simple, modified statistical technique, based around a simple two-group t-test, with error functions and spatial autocorrelation taken into account, finds no differences between the example groups of participants. That conclusion is reinforced by analyzing simulated data with or without true differences, when iMAP/iMAP2 always finds significant differences, irrespective of sample size, whereas the modified method finds significant differences for only the largest sample sizes. Previous research using iMAP/iMAP2 may have come to erroneous conclusions about differences in fixation patterns between groups.