The phase diagram of the strongly correlated Hubbard model with intrinsic spin-orbit coupling on the honeycomb lattice is explored here. We obtain the low-energy effective model describing the spin degree of freedom. The resulting model is then studied by the Schwinger boson and Schwinger fermion approaches. The Schwinger boson method elucidates the boundary between the spin liquid phase and the magnetically ordered phases, Neel order, and incommensurate Neel order. Increasing the strength of the spin-orbit coupling is shown to narrow the width of the spin liquid region. The Schwinger fermion approach sheds further light on the nature of the spin liquid phase. We obtained three different candidates for the spin liquid phase within the mean-field approximation, namely, the gapless spin liquid, topological Mott insulator (fractionalized topological insulator), and chiral spin liquid phases. However, we argue that the gauge fluctuations and the instanton effect may suppress the first two spin liquids, while the chiral spin liquid is stable against gauge fluctuations due to its nontrivial topology.

• Received 28 February 2012

DOI:https://doi.org/10.1103/PhysRevB.85.195126