We present a computationally efficient approach to solve the time-dependent Kohn-Sham equations in real-time using higher-order finite-element spatial discretization, applicable to both pseudopotential and all-electron calculations. To this end, we develop an a priori mesh adaption technique, based on the semi-discrete (discrete in space but continuous in time) error estimate on the time-dependent Kohn-Sham orbitals, to construct a close to optimal finite-element discretization. Subsequently, we obtain the full-discrete error estimate to guide our choice of the time-step. We employ spectral finite-elements along with Gauss-Legendre-Lobatto quadrature to render the overlap matrix diagonal, simplifying the inversion of the overlap matrix that features in the evaluation of the discrete time-evolution operator. We use the second-order Magnus operator as the time-evolution operator in all our calculations. Furthermore, the action of the discrete Magnus operator, expressed as exponential of a matrix, on the Kohn-Sham orbitals is obtained efficiently through an adaptive Lanczos iteration. We observe close to optimal rates of convergence of the dipole moment with respect to spatial and temporal discretization, for both pseudopotential and all-electron calculations. We demonstrate a staggering 100-fold reduction in the computational time afforded by higher-order finite-elements over linear finite-elements. We present comparative studies, in terms of accuracy and efficiency, of our approach against finite-difference based discretization for pseudopotential calculations, and demonstrate significant computational savings when compared to the finite-difference method. We also demonstrate the competence of higher-order finite-elements for all-electron benchmark systems. Lastly, we observe good parallel scalability of the proposed method on many hundreds of processors.