In this paper, we present a numerical model that can describe the pore formation/cavitation in viscoelastic food materials during drying. The food material has been idealized as a spherical object, with a core/shell structure and a central gas-filled cavity. The shell represents a skin as present in fruits/vegetables, having a higher elastic modulus than the tissue, which we approximate as a hydrogel. The gas-filled pore is in equilibrium with the core hydrogel material, and it represents pores in food tissues as present in intercellular junctions. The presence of a rigid skin is a known prerequisite for cavitation (inflation of the pore) during drying. For modeling, we follow the framework of Suo and coworkers, describing the inhomogeneous large deformation of soft materials like hydrogels - where stresses couple back to moisture transport. In this paper, we have extended such models with energy transport and viscoelasticity, as foods are viscoelastic materials, which are commonly heated during their drying. To approach the realistic properties of food materials we have made viscoelastic relaxation times a function of Tg/T, the ratio of (moisture dependent) glass transition temperature and actual product temperature. We clearly show that pore inflation only occurs if the skin gets into a glassy state, as has been observed during the (spray) drying of droplets of soft materials like foods.