The interest in promoting food and water security through development projects has led to the need to evaluate the impact of these projects. This study evaluates the impact from transitioning to a modern irrigation technology. Deciding to adopt or not an alternative irrigation technology (sprinklers) is not necessarily a random determination. Therefore, selection bias can be present and this can lead to biased estimates. In this study, we apply methodological specifications to correct for self-selectivity biases. Then, we measure and compare the technical efficiency scores from adopters and non-adopters. The empirical application uses data covering 56 small-scale greenhouse farms in Crete (Greece) for the cropping years 2009-2013. The results reveal that the average technical efficiency for farmers who adopted sprinkler irrigation is lower than the group of non-adopters, when the presence of selectivity bias cannot be rejected. This implies that either the farmers need more time to incorporate the know-how of the newly acquired technology or they become less motivated after the adoption. Consequently, agricultural water saving technologies need to be promoted in combination with support to the farmers, so they can cope with the lower performance in the first years after adoption.