In this paper Nowadays, online transactions are increasing rapidly and payment scams are also increasing. A hacker is a person who gains access to another person's financial information. Improving feature selection is important to detect transaction risk in large, multidimensional data. Fraud detection and classification performance is divided into two parts: training data and test data. During this process we have focused on data analysis and pre-processing. In a transaction’s dataset, the predictive variables influence how well a machine learning algorithm for transaction risk identification classifies and identifies fraudulent transactions. This study examines the application of machine learning techniques in improving fraud detection in the banking industry. Using advanced algorithms and historical transaction data, the model aims to identify anomalous patterns that indicate fraudulent activity. Feature engineering, ensemble methods, and anomaly detection algorithms are used to improve the accuracy and performance of fraud detection systems. reducing false positives and provides an adaptive security mechanism. By harnessing the power of machine learning, this approach aims to strengthen the banking sectors defines against sophisticated fraud schemes. This study not only contributes to the academic understanding of fraud detection, but also provides industry practitioners with a practical framework for implementing proactive measures. In navigating the complex landscape of financial security, this research serves as a guide for building agile and responsive systems that stay ahead of the threats that evolve in the ever-changing landscape of the banking industry.