Waste biomass pelletization is a suitable process for obtaining energy dense solid fuel. If the pellets are further processed by torrefaction, both their qualitative and energy parameters are changed. Therefore, the objective of this study is to investigate the effect of the torrefaction process on pellets made from coffee grounds, garden chaff and rapeseed straw. Mechanical parameters, such as the pellet durability index, the wettability index of the pellets, their water moisture resistance, hardness, or specific and bulk density were determined for evaluation. Furthermore, pellet energy parameters such as the heat of combustion, calorific value, or elemental analysis were compared. Pelletization of the above-mentioned waste biomass and the torrefaction of the pellets were carried out in pilot plants. The first results showed that the torrefaction process for all samples increased the values of the heat of combustion or the values of the carbon content. The energy value of the pellets increases. The higher heating value of torrefied coffee ground pellets reached 26 MJ kg−1. The ash content also increases with these values. A simple energy balance has been carried out. Therefore, the study shows an insight into the energy use of coffee grounds and chaff from the garden compared to the results of the rapeseed straw pellets for the pelletization and torrefication process.