The passwords for authenticating users are susceptible to shoulder-surfing attacks in which attackers learn users' passwords through direct observations without any technical support. A straightforward solution to defend against such attacks is to change passwords periodically or even constantly, making the previously observed passwords useless. However, this may lead to a situation in which users run out of strong passwords they can remember, or they are forced to choose passwords that are weak, correlated, or difficult to memorize. To achieve both security and usability in user authentication, we propose EvoPass, the first evolvable graphical password authentication system. EvoPass transforms a set of user-selected pass images to pass sketches as user credentials. Users are required to identify their pass sketches from a set of challenge images for user authentication. Particularly, EvoPass improves password strength gradually over time through continually degrading pass sketches without annoying users to reselect pass images. The evolving feature makes it difficult for observational adversaries to identify the pass sketches, even though part of pass sketches may have been exposed to adversaries previously. We introduce two metrics, Information Retention Rate (IRR) and Password Diversity Score (PDS) to guide the process of generating pass sketches and a set of challenge images. Our experimental analysis reveals that applying reasonable IRR and PDS in EvoPass can remarkably improve the resistance to shoulder-surfing attacks without negatively affecting user experience. We also implement a prototype of EvoPass on Android platform with reasonable IRR and PDS applied. Our experimental results on the prototype further demonstrate that EvoPass could work efficiently and achieve a desired usability.