Significant wave height prediction is challenging owing to the nonlinear and nonsmooth attributes of wave heights. This study presents a hybrid model coupling group method of data handling (GMDH) with grey wolf optimizer (GWO) for the prediction of significant wave heights. The datasets were assembled from three different observations, Stations 41001, 41002 and 44004 in the Atlantic; the datasets of Stations 41001 and 41002 were used for training, and those of Station 44004 were used for testing. The performance of the GWO-GMDH model was compared with four artificial intelligence models, the GMDH, gene expression programming (GEP), adaptive neuro-fuzzy inference system (ANFIS) and back propagation (BP) neural network models, and one empirical equation derived by Buckingham π-theorem. Both regression plots and statistical indices (e.g., correlation coefficient (R), root mean squared error (RMSE), mean squared error (MSE) and mean absolute percentage error (MAPE)) were adopted to evaluate the performance of the hybrid GWO-GMDH model. The MSE, RMSE, MAPE and R values were 0.041, 0.202, 7.353% and 0.953, respectively, for the training datasets and 0.031, 0.175, 7.598% and 0.941, respectively, for the testing datasets. Compared with the single GMDH model, the statistical indices of the training datasets of the hybrid GWO-GMDH model were almost the same; however, the MSE, RMSE and MAE values decreased by 24.39%, 13.37% and 7.95%, respectively, and the R value increased by 2.28% in the testing datasets. Compared with the GEP, BP, and ANFIS models and empirical equation models, the GWO-GMDH model also shows high accuracy and robustness, especially compared with empirical formulations. In addition, a graphical user interface (GUI) was developed to facilitate the application of practical engineering use.