In this paper, a novel technique for quantitative vibration analysis using time-average electronic speckle pattern interferometry is proposed. An amplitude-varied time-average refreshing reference frame method is used to capture a fringe pattern with a better fringe contrast than the conventional reference frame technique. The recorded fringe patterns with improved contrast provide better mode shape visibility and are easier to process. A derivative-based regularized phase tracker model is used to retrieve vibration amplitudes from a single fringe pattern. The method does not require a phase shifter to obtain the mode shape or amplitude. The method provides unwrapped amplitude and amplitude derivatives maps directly, so a separate phase unwrapping process is not required. Experimental work is carried out using a circular aluminum plate test specimen and the results are compared with a finite element method modal analysis. Both experimental and numerical results show that the proposed method is robust and accurate.