Correcting the three-dimensional geometric error is essential to effectively use the multi-temporal unmanned aerial vehicle (UAV) orthophoto and digital surface model (DSM) acquired from the agricultural field. Although ground control points (GCPs) obtained through field surveys are usually used to calibrate geometrical errors establishing/maintaining GCPs and surveying them in the field are time-consuming and inefficient. Therefore, we propose a simple and efficient methodology to improve the geometric registration of multi-temporal orthophotos and DSMs without GCPs. In the proposed method, coarse to fine image registration is performed first, which corrects severe to slight errors by sequential feature and area-based matching methods. Subsequently, we extract height-invariant regions in multi-temporal DSM pairs, called elevation invariant feature (EIF), using the EIFs to register DSMs by estimating a linear regression model. Various experiments were conducted to analyze the absolute and relative accuracies using ten multi-temporal orthophotos and DSMs, and the robustness of the proposed method was evaluated using data obtained from another site. The experimental results demonstrate that the geometric quality of registered orthophotos and DSMs was significantly improved.