We use the density matrix renormalization group (DMRG) to study the correlated electron states favored by the Coulomb interaction projected onto the narrow bands of twisted bilayer graphene within a spinless one-valley model. The Hilbert space of the narrow bands is constructed from a pair of hybrid Wannier states with opposite Chern numbers, maximally localized in one direction and Bloch extended in another direction. Depending on the parameters in the Bistritzer-Macdonald model, the DMRG in this basis determines the ground state at one particle per unit cell to be either the quantum anomalous Hall (QAH) state or a state with zero Hall conductivity which is nearly a product state. Based on this form, we then apply the variational method to study their competition, thus identifying three states: the QAH, a gapless $C_2\mathcal{T}$-symmetric nematic, and a gapped $C_2\mathcal{T}$-symmetric stripe. In the chiral limit, the energies of the two $C_2\mathcal{T}$-symmetric states are found to be significantly above the energy of the QAH. However, all three states are nearly degenerate at the realistic parameters of the Bistritzer-Macdonald model. The single-particle spectrum of the nematic contains either a quadratic node or two close Dirac nodes near $\mathrm{\Gamma }$. Motivated by the Landau level degeneracy found in this state, we propose it to be the state observed at the charge neutrality point once spin and valley degeneracies are restored. The optimal period for the $C_2\mathcal{T}$ stripe state is found to be two unit cells. In addition, using the fact that the topological charge of the nodes in the $C_2\mathcal{T}$-nematic phase is no longer described simply by their winding numbers once the translation symmetry is broken, but rather by certain elements of a non-Abelian group that was recently pointed out, we identify the mechanism of the gap opening within the $C_2\mathcal{T}$ stripe state. Although the nodes at the Fermi energy are locally stable, they can be annihilated after braiding with other nodes connecting them to adjacent (folded) bands. Therefore, if the translation symmetry is broken, the gap at one particle per unit cell can open even if the system preserves the $C_2\mathcal{T}$ and valley $\text{U}\left(1\right)$ symmetries, and the gap to remote bands remains open.

• Received 28 February 2020
• Accepted 14 July 2020

DOI:https://doi.org/10.1103/PhysRevB.102.035161