Purpose The purpose of this study is to implement a new class of similarity transformation in analyzing the three-dimensional boundary layer flow of hybrid nanofluid. The Cu-Al2O3/water hybrid nanofluid is formulated using the single-phase nanofluid model with modified thermophysical properties. Design/methodology/approach The governing partial differential equations are reduced to the ordinary (similarity) differential equations using the proposed similarity transformation. The resulting equations are programmed in Matlab software through the bvp4c solver to obtain their solutions. The features of the reduced skin frictions and the velocity profiles for different values of the physical parameters are analyzed and discussed. Findings The non-uniqueness of the solutions is observed for certain physical parameters. The dual solutions are perceived for both permeable and impermeable cases and being the main agenda of the work. The execution of stability analysis proves that the first solution is undoubtedly stable than the second solution. An increase in the mass transpiration parameter leads to the uniqueness of the solution. Oppositely, as the injection parameter increase, the two solutions remain. However, no separation point is detected in this problem within the considered parameter values. The present results are decisive to the pair of alumina and copper only. Originality/value The present findings are original and can benefit other researchers particularly in the field of fluid dynamics. This study can provide a different insight of the transformation that is applicable to reduce the complexity of the boundary layer equations.