Based on the discrete element method (DEM), four packed beds composed of soybean kernels with diameters of 6.4, 6.8, 7.4 mm, and the mixture of three kinds of particles were established. Then, a double-diffusion heat and mass transfer model between the grain pile and the interstitial air was established based on the local mass and thermal non-equilibrium (LMTNE) mechanism. Finally, employing particle-resolved computational fluid dynamics (PRCFD), the heat and mass transfer between the grain kernels and air during the drying process in the four packed beds were numerically resolved. It was found that the packed bed formed by stacking particles of different diameters had a minimum porosity of 0.4547. The radial porosity of the packed bed oscillates and decreases toward the central axis, while the tortuosity of the airflow path oscillates and decays toward the periphery. The mass transfer Biot number for soybean kernels with diameters of 6.4, 6.8, and 7.4 mm were 2.38 × 106, 2.44 × 106, and 2.53 × 106, respectively. This indicates that the mass transfer rate in the grain pile primarily depends on the magnitude of the moisture diffusion coefficient within the grain kernels. Compared with mass diffusion, thermal diffusion occurs much faster, which results in temperature gradients in packed beds only existing in the first 5 min of drying, and the drying rate in the early stage is higher than that in the later stage. Most importantly, the airflow characteristics, heat and moisture content are not in local equilibrium in the packed bed, and should be considered when designing drying systems.