The performance of a variational regularization technique to improve robustness of inverse treatment planning for intensity modulated radiotherapy is analyzed and tested. Inverse treatment planning is based on the numerical solutions to the Fredholm integral equation of the first kind which is ill-posed. Therefore, a fundamental problem with inverse treatment planning is that it may exhibit instabilities manifested in nonphysical oscillations in the beam intensity functions. To control the instabilities, we consider a variational regularization technique which can be applied for the methods which minimize a quadratic objective function. In this technique, the quadratic objective function is modified by adding of a stabilizing functional that allows for arbitrary order regularization. An optimal form of stabilizing functional is selected which allows for both regularization and good approximation of beam intensity functions. The regularized optimization algorithm is shown, by comparison for a typical case of a head-and-neck cancer treatment, to be significantly more accurate and robust than the standard approach, particularly for the smaller beamlet sizes.