Timing noise in the data on accretion-powered millisecond pulsars (AMP) appears as irregular pulse phase jumps on timescales from hours to weeks. A large systematic phase drift is also observed in the first discovered AMP SAX J1808.4-3658. To study the origin of these timing features, we use here the data of the well studied 2002 outburst of SAX J1808.4-3658. We develop first a model for pulse profile formation accounting for the screening of the antipodal emitting spot by the accretion disk. We demonstrate that the variations of the visibility of the antipodal spot associated with the receding accretion disk cause a systematic shift in Fourier phases, observed together with the changes in the pulse form. We show that a strong secondary maximum can be observed only in a narrow intervals of inner disk radii, which explains the very short appearance of the double-peaked profiles in SAX J1808.4-3658. By directly fitting the pulse profile shapes with our model, we find that the main parameters of the emitting spot such as its mean latitude and longitude as well as the emissivity pattern change irregularly causing small shifts in pulse phase, and the strong profile variations are caused by the increasing inner disk radius. We finally notice that significant variations in the pulse profiles in the 2002 and 2008 outbursts of SAX J1808.4-3658 happen at fluxes differing by a factor of 2, which can be explained if the inner disk radius is not a simple function of the accretion rate, but depends on the previous history.