Generalized XY model with competing antiferromagnetic and antinematic interactions.

I study effects of q-order antinematic (AN_{q}) interactions on the critical behavior of the antiferromagnetic (AF) XY model on a square lattice. It is found that the evolution of the phase diagram topology of such AF-AN_{q} models with the parameter q does not follow the same line as for the corresponding FM-N_{q} models with the ferromagnetic (FM) and q-order nematic (N_{q}) interactions. Their phase diagrams are similar only for odd values of the parameter q. In such cases the respective phases reported in the FM-N_{q} models are observed in the AF-AN_{q} models on each of the two AF-coupled sublattices and the corresponding phase transitions are concluded to be of the same kind. On the other hand, for even values of q the phase diagrams of the AF-AN_{q} models are different from the FM-N_{q} models and their topology does not change with q. Besides the pure AF and AN_{q} phases, observed at higher temperatures in the regions of the dominant respective couplings, at low temperatures there is a new canted (C)AF phase, which results from the competition between the AF and AN_{q} ordering tendencies and has no counterpart in the FM-N_{q} model. The phase transitions to the CAF phase from both AF and AN_{q} phases appear to be of the Berezinskii-Kosterlitz-Thouless nature.