The stated goal of this study is to comprehensively understand the dynamic instability characteristics of thin nanoplate structures, taking into account the combined effects of both size and surface energy. To achieve this, we present a computational approach based on the non-classical Kirchhoff plate theory within the framework of NURBS-based isogeometric analysis (IGA). The modified couple stress theory, which includes one material length scale parameter, is implemented to adequately account for the size-dependent effect. Additionally, the Gurtin–Murdoch surface elasticity is effectively utilized to assess the influence of surface energy. For the first time, we explore the dynamic instability regions of thin nanoplates, accounting for both size and surface energy effects simultaneously. To obtain solutions from the Mathieu-Hill equations, the Bolotin method is applied in this research. A number of numerical investigations on square, circular and annular thin nanoplate models under periodic in-plane compressive loads are conducted to assess the influence of several crucial input factors on the regions of dynamic instability. The present outcomes reveal that both size and surface energy effects have a remarkable influence on the dynamic instability characteristics of the nanoplates. Accordingly, these effects contribute to an enhancement in structural rigidity at the small-scale level. Furthermore, we explore that in the case of ultra-thin nanoplates, the influence of surface energy becomes dominant. The obtained findings provide valuable insights into the behavior of thin nanoplates under dynamic in-plane loading conditions as well as can contribute to the design of more robust engineering structures.