AbstractSmart home is the main part of smart intelligent system here the remote users share the sensitive information through an insecure medium to access such smart devices, which becomes security issues. The recent user authentication protocols have used to solve those problems and provide secure communication. Consumer traffic increase the risk of illegal user as legal user and radio channels are extra vulnerable to listeners. For further security enhancement, we proposed an optimal cluster based remote user authentication (OCRUA) protocol for smart home environment using hybrid soft computing techniques. The first contribution of proposed protocol is to introduce squirrel induced butterfly optimization (SBO) algorithm for cluster formation, which groups the smart devices. Then, we compute the cluster head (CH) using the teacher learning based deep neural network (TL‐DNN) based on multiple design constraints. The second contribution is to illustrate remote user authentication using optimal elliptic curve cryptography (OECC) which encrypts the sensitive information before forward to gateway. At long last, the concert of planned OCRUA protocol evaluates use different replication scenarios and shows the effectiveness over the existing state‐of‐art protocols.