This project explores the application of Convolutional Neural Networks in Handwritten digit recognition. Exploiting the widely used MNIST dataset, we proposed a deep learning model that encompasses the Ensemble model that is capable of predicting and recognizing the handwritten digits and achieves remarkable precision. The dataset contains 70000 images in total, where 55000 images are for training, 5000 images for validating, and 10000 images for testing. By using CNN, we surpassed the traditional way of training the algorithm with machine learning. By incorporating robust pre-processing techniques and innovative training ideas and strategies, our model showcases resilience to real-world problems and challenges which includes reading the postal codes in the written mail, reading the digits in the handwritten checks, digits drawn on the mobile touch panel, etc. To better extract the features of the complex handwritten digits, bagging is used from the Ensemble techniques. The objective of this project is to find a better optimizer to work with ensemble models, which encompasses the merging of optimizers with the deep learning models. Key Words: Handwritten digits, CNN, Ensemble Model, Optimizers, MNIST.