In this article we study the existence question of Anosov diffeomorphisms on an infra-nilmanifold. After establishing a general existence criterion in terms of the associated holonomy representation, we concentrate on infra-nilmanifolds for which the covering Lie group is a free nilpotent Lie group. In turns out that in this case the criterion obtained before can be reduced drastically. Finally, we completely solve the existence question in case the covering Lie group is free 2-step nilpotent and the holonomy group is abelian.