We investigate complex spin structures of frustrated two-dimensional Cr, Mn, and Fe monolayer magnets on a triangular lattice provided by the Cu(111) substrate. First we establish a zero-temperature phase diagram of possible spin structures on the basis of the classical Heisenberg model up to the third-nearest neighbor exchange interaction. Second we carried out first-principles total energy calculations on the basis of the vector-spin density formulation of the density functional theory using the full potential linearized augmented plane wave (FLAPW) method in film geometry for a set of complex non-collinear spin structures. We found, the ground state of Fe is ferromagnetic, Cr exhibits a coplanar, two-dimensional non-collinear 120 Néel state and Mn a three-dimensional non-collinear ground state, the 3Q-state. Incommensurate spin-spiral states are expected for a FeMn alloy on Cu(111). We employ the constrained local moment method to estimate the exchange parameters of the model Hamiltonians. We show that for Mn higher-order spin interactions are the origin of the 3Q-state for Mn. The combination of ab initio calculations and model Hamiltonians provides a powerful tool to investigate the magnetic structures of complex magnetic systems.