With the rising integration of IoT devices within smart home environments, securing these interconnected systems against unauthorized access and cyber threats has become increasingly critical. This study introduces a novel methodology utilizing an advanced Artificial Neural Network (ANN) frame- work to enhance attack and anomaly detection capabilities within smart home networks. The objective is to develop a robust model that outperforms traditional detection methods and provides high accuracy and low false positive rates in identifying potential security threats. The research employs a pioneering approach to train the ANN by incorporating a Fractional Stochastic Gradient Descent optimizer grounded in Grunwald–Letnikov fractional calculus. This method was chosen for its potential to refine learning processes and improve detection accuracy over standard optimizers. The evaluation of the model was performed using the DS2OS traffic traces dataset, applying precision, sensitivity (recall), and specificity metrics to assess performance. The proposed model demonstrated exceptional performance with an accuracy of 0.9951. It significantly surpassed traditional methods like Logistic Regression and Support Vector Machines. The precision achieved was high, indicating a low rate of false pos- itives, while the sensitivity and specificity values underscore the model’s ability to identify both typical and unconventional behaviours within the network accurately. This study introduces a robust and efficient ANN-based methodology for enhancing security in smart home IoT networks. Using a fractional stochastic gradient descent optimizer has proven effective in improving the model’s accuracy and reliability in detecting anomalies and attacks. The findings suggest significant implications for the future of IoT security, highlighting the potential for broader applications of fractional calculus in machine learning to enhance cybersecurity measures in various domains.