Given a graph G, a set S of vertices in G is a general position set if no triple of vertices from S lie on a common shortest path in G. The general position achievement/avoidance game is played on a graph G by players A and B who alternately select vertices of G. A selection of a vertex by a player is a legal move if it has not been selected before and the set of selected vertices so far forms a general position set of G. The player who picks the last vertex is the winner in the general position achievement game and is the loser in the avoidance game. In this paper, we prove that the general position achievement/avoidance games are PSPACE-complete even on graphs with diameter at most 4. For this, we prove that the misère play of the classical Node Kayles game is also PSPACE-complete. As positive results, we obtain linear time algorithms to decide the winning player of the general position avoidance game in rook's graphs, grids, cylinders, and lexicographic products with complete second factors.