Downdraft gasifier is a promising technology converting biomass into synthesis gas (syngas) for decentralized heat and power generation. As biomass is seasonal variation and diversity, the design of the gasifier should consider a variety of biomass feedstock. This study investigated a single-designed downdraft gasifier of different biomass pellets using computational fluid dynamics and optimized for producer gas quality and gasifier efficiencies. Three biomass pellets, i.e., wood, bagasse, and palm-oil empty fruit bunch (PEFB), were considered the primary qualitative variable in addition to the main process variables, i.e., air-inlet temperature and equivalence ratio (ER). Response surface methodology (RSM) and multi-objective genetic algorithm (MOGA) were carried out for process and biomass optimization. Tar yield is considered a response for optimization along with carbon conversion (CCE) and cold gas efficiencies (CGE). The results showed that ER and biomass type are crucial. The range of ER for each biomass should be carefully selected. The maximum efficiencies of the gasifier can be achieved using PEFB pellets, whereas the minimum tar yield was attained using wood pellets. The CCE and CGE of 93.11% and 65.16%, respectively, and the tar yield of 12.36 mg Nm−3, were obtained at an optimum ER of 0.28 and air-inlet temperature of 1272.8 K using PEFB pellets.