The relevance of selecting an appropriate bed material in fluidized bed gasification is a crucial aspect that is often underestimated. The ideal material should be economical, resistant to high temperatures and have small chemical interaction with biomass. However, often only the first of such three aspects is considered, neglecting the biomass–bed interaction effects that develop at high temperatures. In this work, olivine and K-feldspar were upscale-tested in a prototype fluidized bed gasifier (FBG) using arboreal biomass (almond shells). The produced syngas in the two different tests was characterized and compared in terms of composition (H2, CH4, CO, CO2, O2) and fate of contaminants such as volatile organic compounds (VOCs), tar and metals.. Moreover, the composition of olivine and K-feldspar before and after the biomass gasification process has been characterized. The aim of this work is to show which advantages and disadvantages there are in choosing the most suitable material and to optimize the biomass gasification process by reducing the undesirable effects, such as heavy metal production, bed agglomeration and tar production, which are harmful when syngas is used in internal combustion engines (ICE). It has been observed that metals, such as Ni, Cu, Zn, Cd, Sn, Ba and Pb, have higher concentrations in the syngas produced by using olivine as bed material rather than K-feldspar. In particular, heavy metals, such as Pb, Cu, Cd, Ni and Zn, show concentrations of 61.06 mg/Nm3, 15.29 mg/Nm3, 17.97 mg/Nm3, 37.29 mg/Nm3 and 116.39 mg/Nm3, respectively, compared to 23.26 mg/Nm3, 11.82 mg/Nm3, 2.76 mg/Nm3, 24.46 mg/Nm3 and 53.07 mg/Nm3 detected with K-feldspar. Moreover, a more hydrogen-rich syngas when using K-feldspar was produced (46% compared to 39% with olivine).