We study the strangeness $S=-2$ baryon-baryon interactions in relativistic chiral effective field theory at leading order. Among the 15 relevant low energy constants, eight of them are determined by fitting to the state of the art lattice QCD data of the HAL QCD Collaboration (with $m_\pi=146$ MeV), and the rest are either taken from the study of the $S=-1$ hyperon-nucleon systems, assuming strict SU(3) flavor symmetry, or temporarily set equal to zero. Using the so-obtained low energy constants, we extrapolate the results to the physical point, and show that they are consistent with the available experimental scattering data. Furthermore, we demonstrate that the $\Lambda\Lambda$ and $\Xi N$ phase shifts near the $\Xi N$ threshold are very sensitive to the lattice QCD data fitted, to the pion mass, and to isospin symmetry breaking effects. As a result, any conclusion drawn from lattice QCD data at unphysical pion masses (even close to the physical point) should be taken with caution. Our results at the physical point, similar to the lattice QCD data, show that a resonance/quasi-bound state may appear in the $I=0$ $\Lambda\Lambda$/$\Xi N$ channel.