Eye movement (EM) is considerably a new behavioral modality for biometric authentication. In this work, we use this modality in the context of continuous driver authentication. Existing models rely on different modalities that limit their usage or are inconvenient to drivers. We propose an end-to-end learning model that takes the remote eye movement profiles solely and produces embeddings for driver authentication scenarios. The model is based on Long short-term memory (LSTM) and dense networks to learn temporal characteristics from the EM profiles. We focus on low-rate devices because of their affordability. Yet, they present a challenge because of their limited ability to capture quality measurements. To evaluate our model, two low frame-rate devices are used to build our datasets, which are Autocruis and GazePoint. The authentication performance outperforms state-of-the-art with as low as 30 seconds frame length with both devices. The best authentication performances for the identification/verification modes are 92.38/0.76% and 91.05/0.11% for the first and the second datasets, respectively.