Given a smooth projective algebraic surface X, a point O in X and a big divisor D on X, we consider the set of all Newton-Okounkov bodies of D with respect to valuations of the field of rational functions of X centred at O, or, equivalently, with respect to a flag (E,p) which is infinitely near to O, in the sense that there is a sequence of blowups mapping the smooth, irreducible rational curve E to O. The main objective of this paper is to start a systematic study of the variation of these infinitesimal Newton-Okounkov bodies as (E, p) varies, focusing on the case X = P2.