In this paper, new mathematical model is developed for computing the rectangular windowed low-pass FIR filter transfer function where the rotation angle of fractional Fourier transform is used as a free parameter. By changing the rotation angle in the range from \(0\) to \(\pi /2\), the transition bandwidth and the stop-band attenuation can be tuned. The closed-form expression derived establishes a direct relationship between magnitude response of the filter and fractional angle of the transform. It is shown that sharp transition bandwidth followed with an increase in the stop-band attenuation can be achieved with the rectangular window for small values of the rotation angle. Digital filters with variable frequency responses can be obtained directly by raising the filter length. This involves recomputation of impulse response coefficients and redesigning of a new filter from the existing one. More number of data registers, multipliers and filter coefficients will be needed, which adds to computational burden. Using proposed fractional Fourier transformation technique, the response characteristics of FIR filter can be tuned during operation by changing only one parameter, while keeping coefficients of the fixed FIR filter unchanged.