This paper deals with the analysis of various operating conditions of power transformer such as energization, inrush current, over excitation and CT saturation. In this work a new relaying algorithm is used to enhance the fault detection sensitivities of conventional techniques by using a fuzzy logic approach. An improved differential protection scheme is proposed which uses voltages and current to detect the abnormalities in the different zone of protection. The magnetizing inrush current during transformer energization is taken with a simplified excitation curve. Discrete fourier transform is used for extraction of fundamental and higher harmonic components of the differential current. The fuzzy based protective relaying algorithm will analyze inrush current, residual flux, saturation flux, harmonics restraint and percentage differential characteristics curve. This method of fuzzy based protective relaying algorithm is tested with MATLAB simulink model and a performance comparison of conventional scheme and fuzzy logic based protection scheme is provided.