Recent cosmological hydrodynamical simulations suggest that integral field spectroscopy can connect the high-order stellar kinematic moments h3 (~skewness) and h4 (~kurtosis) in galaxies to their cosmological assembly history. Here, we assess these results by measuring the stellar kinematics on a sample of 315 galaxies, without a morphological selection, using 2D integral field data from the SAMI Galaxy Survey. A proxy for the spin parameter ($\lambda_{R_e}$) and ellipticity ($\epsilon_e$) are used to separate fast and slow rotators; there exists a good correspondence to regular and non-regular rotators, respectively, as also seen in earlier studies. We confirm that regular rotators show a strong h3 versus $V/\sigma$ anti-correlation, whereas quasi-regular and non-regular rotators show a more vertical relation in h3 and $V/\sigma$. Motivated by recent cosmological simulations, we develop an alternative approach to kinematically classify galaxies from their individual h3 versus $V/\sigma$ signatures. We identify five classes of high-order stellar kinematic signatures using Gaussian mixture models. Class 1 corresponds to slow rotators, whereas Classes 2-5 correspond to fast rotators. We find that galaxies with similar $\lambda_{R_e}-\epsilon_e$ values can show distinctly different h3-$V/\sigma$ signatures. Class 5 objects are previously unidentified fast rotators that show a weak h3 versus $V/\sigma$ anti-correlation. These objects are predicted to be disk-less galaxies formed by gas-poor mergers. From morphological examination, however, there is evidence for large stellar disks. Instead, Class 5 objects are more likely disturbed galaxies, have counter-rotating bulges, or bars in edge-on galaxies. Finally, we interpret the strong anti-correlation in h3 versus $V/\sigma$ as evidence for disks in most fast rotators, suggesting a dearth of gas-poor mergers among fast rotators.