Evidence for a Phase Transition in the Quantum Spin Liquid State of a Kitaev Candidate α-RuCl3

Abstract

The Kitaev quantum spin liquid (QSL) on the two-dimensional honeycomb lattice epitomizes an entangled topological state, where the spins fractionalize into Majorana fermions. This state has aroused tremendous interest because it harbors non-Abelian anyon excitations. The half-integer quantized thermal Hall (HIQTH) conductance observed in $\alpha$-RuCl$_3$ is a key signature of these excitations. However, the fate of this topologically nontrivial state at intense fields remains largely elusive. Here, we report the thermal conductivity $\kappa$ and specific heat $C$ of $\alpha$-RuCl$_3$ with in-plane magnetic fields $H$. For the field direction perpendicular to the Ru-Ru bond, where the HIQTH effect is observed, we find a discontinuous jump in $\kappa(H)$ and a peak anomaly in $C(H)$ at $\mu_0H^*\approx11$\,T, evidencing a weak first-order phase transition. Remarkably, the HIQTH effect vanishes close to $H^*$. Furthermore, we find that the spin-fractionalization feature is retained well above $H^\ast$. These imply the emergence of the phase transition that separates two QSL phases with distinct topological properties.