Pneumonia is a dangerous disease that kills millions of children and elderly patients worldwide every year. The detection of pneumonia from a chest x-ray is perpetrated by expert radiologists. The chest x-ray is cheaper and is most often used to diagnose pneumonia. However, chest x-ray-based diagnosis requires expert radiologists which is time-consuming and laborious. Moreover, COVID-19 and pneumonia have similar symptoms which leads to false positives. Machine learning-based solutions have been proposed for the automatic prediction of pneumonia from chest X-rays, however, such approaches lack robustness and high accuracy due to data imbalance and generalization errors. This study focuses on elevating the performance of machine learning models by dealing with data imbalanced problems using data augmentation. Contrary to traditional machine learning models that required hand-crafted features, this study uses transfer learning for automatic feature extraction using Xception and VGG-16 to train classifiers like support vector machine, logistic regression, K nearest neighbor, stochastic gradient descent, extra tree classifier, and gradient boosting machine. Experiments involve the use of hand-crafted features, as well as, transfer learning-based feature extraction for pneumonia detection. Performance comparison using Xception and VGG-16 features suggest that transfer learning-based features tend to show better performance than hand-crafted features and an accuracy of 99.23% can be obtained for pneumonia using chest X-rays.