We consider layered decorated honeycomb lattices at two-thirds filling, as realized in some trinuclear organometallic complexes. Localized $S=1$ moments with a single-spin anisotropy emerge from the interplay of Coulomb repulsion and spin molecular-orbit coupling (SMOC). Magnetic anisotropies with bond-dependent exchange couplings occur in the honeycomb layers when the direct intracluster exchange and the spin molecular-orbital coupling are both present. We find that the effective spin exchange model within the layers is an XXZ + ${120}^{\circ }$ honeycomb quantum compass model. The intrinsic nonspherical symmetry of the multinuclear complexes leads to very different transverse and longitudinal spin molecular-orbital couplings, which greatly enhances the single-spin and exchange coupling anisotropies. The interlayer coupling is described by an XXZ model with anisotropic biquadratic terms. As the correlation strength increases the system becomes increasingly one-dimensional. Thus, if the ratio of SMOC to the interlayer hopping is small this stabilizes the Haldane phase. However, as the ratio increases there is a quantum phase transition to the topologically trivial “$D$ phase.” We also predict a quantum phase transition from a Haldane phase to a magnetically ordered phase at sufficiently strong external magnetic fields.

• Received 24 March 2017
• Revised 27 September 2017

DOI:https://doi.org/10.1103/PhysRevB.96.205118