In molecular-based quantum-spin-liquid candidate EtMe$_3$Sb[Pd(dmit)$_2$]$_2$ with two-dimensional $S$=1/2 triangular lattice, a finite residual linear term in the thermal conductivity, $\kappa_0/T\equiv\kappa/T (T \rightarrow 0)$, has been observed and attributed to the presence of itinerant gapless excitations. Here we show that the data of $\kappa$ measured in several single crystals are divided into two groups with and without the residual linear term. In the first group with finite $\kappa_0/T$, the phonon thermal conductivity $\kappa_{ph}$ is comparable to that of other organic compounds. In these crystals, the phonon mean free path $\ell_{ph}$ saturates at low temperatures, being limited by sample size. On the other hand, in the second group with zero $\kappa_0/T$, $\kappa_{ph}$ is one order of magnitude smaller than that in the first group, comparable to that of amorphous solids. In contrast to the first group, $\ell_{ph}$ shows a glassy-like non-saturating behavior at low temperatures. These results suggest that the crystals with long $\ell_{ph}$ are required to discuss the magnetic excitations by thermal conductivity measurements.