Effects of the interplay of neighboring couplings on the possible phase transition of a two-dimensional antiferromagnetic system.

We investigate the phase transition of the quantum spin-1 anisotropic antiferromagnet on a square lattice. The model is described by the Heisenberg Hamiltonian with the nearest-neighbor coupling strengths J_{1a} and J_{1b} along the x and y directions, respectively, and next-nearest-neighbor coupling J_{2}. This model allows Néel state (AF1) and collinear state (AF2). The effects of the spatial and single-ion anisotropy on phase transformation between these two states are explored. Our results show that the two states can exist and have the same critical temperature at D>0 as long as J_{2}=J_{1b}/2. Under such parameters, a first-order phase transformation between these two states below the Néel point can occur when J_{1b} value is not very small and D value is within a narrow range. For J_{2}≠J_{1b}/2, although both states may exist, their Néel temperatures differ. If the Néel point of the AF1 (AF2) state is larger, then at very low temperature, the AF1 (AF2) state is more stable. Thus, in an intermediate temperature, a first-order phase transition between these two states may occur.