Unique features of the nonmagnetic insulator phase are revealed, and the phase diagram of the $t\text{−}{t}^{\prime }$ Hubbard model containing the diagonal transfers $t^{\prime }$ on a square lattice is presented. Using the path-integral renormalization group method, we find an antiferromagnetic phase for small next-nearest-neighbor transfer $t^{\prime }$ and a stripe (or collinear) phase for large $t^{\prime }$ in the Mott insulating region of the strong on-site interaction $U$. For intermediate $t^{\prime }∕t\sim 0.7$ at large $U∕t>7$, we find a longer-period antiferromagnetic-insulator phase with $2\times 4$ structure. In the Mott insulating region, we also find a quantum spin liquid (in other words, a nonmagnetic insulator) phase near the Mott transition to paramagnetic metals for the $t\text{−}{t}^{\prime }$ Hubbard model on the square lattice as well as on the anisotropic triangular lattice. Correlated electrons often crystallize to the Mott insulator usually with some magnetic orders, whereas the “quantum spin liquid" has been a long-sought issue. We report numerical evidence that a nonmagnetic insulating phase gets stabilized near the Mott transition with remarkable properties: The two-dimensional Mott insulators on geometrically frustrated lattices contain a phase with gapless spin excitations and degeneracy of the ground state in the whole Brillouin zone of the total momentum. The obtained vanishing spin renormalization factor suggests that spin excitations do not propagate coherently in contrast to conventional phases, where there exist either magnons in symmetry-broken phases or particle-hole excitations in paramagnetic metals. It imposes a constraint on the possible pictures of quantum spin liquids and supports an interpretation for the existence of an unconventional quantum liquid. The present concept is useful in analyzing a variety of experimental results in frustrated magnets including organic BEDT-TTF compounds and ${}^3\mathrm{He}$ atoms adsorbed on graphite.

• Received 10 March 2006

DOI:https://doi.org/10.1103/PhysRevB.74.014421