We revise the theory of the indirect exchange interaction between magnetic impurities beyond the linear response theory to establish the effect of impurity resonances in the surface states of a three-dimensional topological insulator. The interaction is composed of isotropic Heisenberg, anisotropic Ising, and Dzyaloshinskii-Moriya types of couplings. We find that all three contributions are finite at the Dirac point, which is in stark contrast to the linear response theory which predicts a vanishing Dzyaloshinskii-Moriya-type contribution. We show that the spin-independent component of the impurity scattering can generate large values of the Dzyaloshinskii-Moriya-type coupling in comparison with the Heisenberg and Ising types of couplings, while these latter contributions drastically reduce in magnitude and undergo sign changes. As a result, both collinear and noncollinear configurations are allowed magnetic configurations of the impurities.

• Received 24 January 2018

DOI:https://doi.org/10.1103/PhysRevB.97.180402