The three-dimensional S=1 axial next-nearest neighbour Ising (ANNNI) model is discussed. In addition to the ferromagnetic intra- and inter-layer interactions J0 and J1, two types of competing antiferromagnetic interactions are considered between next-nearest layers, i.e. the ordinary two-site two-spin interaction, J2SiS1+2, ((2-2) model) or three-site four-spin interaction, J3SiSi+12Si+2, ((3-4) model). For both models, the ground state is rigorously obtained by means of the transfer matrix method, and proved to change from ferromagnetic to antiphase spin structure at J2 (or J3)/J1=-1/2 with the increase of competition. From magnetization, internal energy, specific heat and 'absolute magnetization' calculated by the Monte Carlo simulation with Fourier transformation, phase boundaries among paramagnetic, ferromagnetic and modulated phases are determined with the location of the Lifshitz point. It is confirmed that for the (3-4) model the self-spin correlation, Si+12, included in three-site four-spin interaction weakens the frustration between J1 and J3 at high temperatures. In the vicinity of the paramagnetic phase transition, the correlation along the z-direction becomes considerably weaker than the correlation in xy-plane owing to the frustration along the z-direction. Consequently, for both (2-2) and (3-4) model there exists the temperature region in which the system behaves as quasi two-dimensional system.