We study quantum phase transitions in the honeycomb Kitaev model under a magnetic field, focusing on the topological nature of Majorana fermion excitations. We find a gapless phase between the low-field gapless quantum spin liquid and the high-field gapped forced-ferromagnetic state for the antiferromagnetic Kitaev model in the [001] field by using the Majorana mean-field theory, in conjunction with the exact diagonalization and the spin-wave theory supporting the validity of this approach. The transition between the two gapless phases is driven by a topological change of the Majorana spectrum—line node formation interconnecting two Majorana cones. The peculiar change of the Majorana band topology is rationalized by a sign change of the effective Kitaev coupling by the magnetic field, which does not occur in the ferromagnetic Kitaev case. Upon tilting the magnetic field away from [001], the two gapless phases become gapped and topologically nontrivial, characterized by nonzero Chern numbers with different signs. The sign change of the Chern number leads to a reversal of the thermal edge current in the half-quantized thermal Hall effect.

• Received 7 May 2018

DOI:https://doi.org/10.1103/PhysRevB.98.060416