Embedded systems execute applications that execute hardware differently depending on the computation task, generating time-varying workloads. Energy minimization can be reached by using the low-power central processing unit (CPU) frequency for each workload. We propose an autonomous and online approach, capable of reducing energy consumption from adaptation to workload variations even in an unknown environment. In this approach, we improved the AEWMA algorithm into a new algorithm called AEWMA-MSE, adding new functionality to detect workload changes and demonstrating why it is better to use statistical analysis for real user cases in a mobile environment. Also, a new power model for mobile devices based on k-NN algorithm for regression was proposed and validated proving to have a better trade-off between execution time and precision than neural networks and linear regression-based models. AEWMA-MSE and the proposed power model are integrated into a novel algorithm for energy management based on reinforcement learning that suitably selects the appropriate CPU frequency based on workload predictions to minimize energy consumption. The proposed approach is validated through simulation by using real smartphone data from an ARM Cortex A7 processor used in a commercial smartphone. Our proposal proved to have an improvement in the Q-learning cost function and can effectively minimize the average energy consumption by 21% and up to 29% when compared to the already existing approaches.