Brain tumor happens due to the instant and uncontrolled cell growth. It may lead to death if not cured at an early stage. In spite of several promising results and substantial efforts in this research area, the real challenge is to provide the accurate classification and segmentation. The key issue in brain tumor detection develops from the irregular changes in the tumor size, shape and location. In assessing the MRI images, computer-aided diagnoses are playing an extraordinary role and can help clinicians/radiologist. Nowadays, brain tumor has become the most incursive ailment that leads to a very short life expectancy when it reaches its highest grade. This research paper has created a new model using histogram of gradient (HOG) based neural features from MRI images for tumors detection. This research has conducted the feature optimization approach to achieve additional instinctive features from the complex feature vector. We developed a Modified ResNet50 model with HOG technique. The modified ResNet50 model can accurately extract the deep feature using deep learning approach. This model is applied along with the upgraded layered architecture in order to keep the optimal computational efficiency. We have also used the augmentation and feature extraction techniques using machine learning-based ensemble classifier that further provides the optimized fusion vector to identify the tumor. Such hybrid approach provides excellent performance with the detection accuracy of 88% with HOG and modified ResNet50. The results are also compared with the recent state of art methods.