The 120° ordered antiferromagnetic state of the Hubbard model on a triangular lattice presents an interesting case of $U$-controlled competing interactions and frustration. The spin stiffness is found to vanish at $U\approx 6$ due to competing spin couplings generated at finite $U$. The loss of magnetic order due to divergent quantum fluctuations yields a magnetic quantum phase transition in the insulating state at $U\gtrsim 6$. Implications of the quantum spin disordered insulator to the spin-liquid state and Mott transition in the organic systems $\kappa \text{−}{\left(\mathrm{BEDT}\text{−}\mathrm{TTF}\right)}_{2}X$ are discussed. Effects of hole and electron doping on magnetic ordering and spin stiffness are also examined.

• Received 8 November 2004

DOI:https://doi.org/10.1103/PhysRevB.71.214406