Wearable sensor-based gait recognition has received much interest because it is unobtrusive and is user friendly. Many research has been carried out in this area but conventional gait recognition methods are not free from drawbacks. In this paper, accelerometer and gyroscope signals representing gait movements are encoded using covariance matrices. The covariance matrices provide a compact and descriptive representation for the accelerometer and gyroscope signals. Non-singular covariance matrices are inherently Symmetric Positive Define (SPD) matrices. Interpreting such SPD matrices as points on the Riemannian manifold leads to increased performance. However, direct geodesic distance calculation for the matrix manifold may yield a suboptimal result. The proposed method solves this issue by embedding the manifold valued points to a higher dimensional Reproducing Kernel Hilbert Space (RKHS) via Positive Definite Gaussian Kernel functions. Extensive experiments have been conducted on three challenging benchmark datasets and a self-collected dataset. Experiment results testify the performance of the proposed RKHS embedding approach.