The layered LiMO2 (M = Co, Ni, and Mn) materials are commonly used as the cathode materials in the lithium-ion battery due to the distinctive layer structure for lithium extraction and insertion. Although their electrochemical properties have been extensively studied, the structural and magnetic properties of LiNiO2 are still under considerable debate, and the magnetic properties of monoclinic LiMnO2 are seldom reported. In this work, a detailed study of LiNiO2, LiMnO2, and a half-doped material LiNi0.5Mn0.5O2 is performed via both first-principles calculations and Monte Carlo simulations based on the effective spin Hamiltonian model. Through considering different structures, it is verified that a structure with a zigzag-type pattern is the most stable one of LiNiO2. Moreover, in order to figure out the magnetic properties, the spin exchange interactions are calculated, and then magnetic ground states are predicted in these three systems. The results show that LiNiO2 forms a spiral order that is caused by the competition from both the short-range and long-range spin exchange interactions, whereas the magnetic ground state of LiMnO2 is collinearly antiferromagnetic due to its nearest and next-nearest neighbor antiferromagnetic spin exchange interactions. However, LiNi0.5Mn0.5O2 is collinearly ferrimagnetic because of the ferromagnetic nearest neighbor Ni-Ni and Mn-Mn exchange interactions. Our work demonstrates the competition between the different exchange interactions in these cathode materials, which may be relevant to the performance of the lithium-ion battery.