Retinal image registration is clinically significant to help clinicians obtain more complete details of the retinal structure by correlating the properties of the retina. However, existing methods suffer from great challenges due to time-consuming optimization and lack of ground truth. In this paper, we propose an unsupervised learning framework for non-rigid retinal image registration, which directly learns the mapping from a retinal image pair to their corresponding deformation field without any supervision such as ground truth registration fields. Specifically, we formulate the complex mapping as a parameterized deformation function, which can be represented and optimized by a deep neural network. Furthermore, the Structure-Driven Regression Network (SDRN) framework is applied to compute the multi-scale similarity combined with contextual structures (e.g., vessel distribution, optic disk appearance, and edge information) to guide the end-to-end learning procedure more effectively with unlabeled data. Given a new pair of images, our method can quickly register images by directly evaluating the parametric function using the learned parameters, which runs faster than traditional registration algorithms. Experimental results, performed on the public challenging dataset (FIRE), show that our method achieves an average Dice similarity coefficient (DSC) of 0.753 with short execution times (0.021 s), which is more accurate and robust than existing approaches and promises to significantly speed up retinal image analysis and processing.