Roller screens play a decisive character in the efficiency of pelletizing circuits and induration furnaces. Unlike vibrating screens, there have been no studies on roller screens to determine their capacity and dimensions. Most studies have utilized the discrete element method (DEM) simulation to assess roller screen performance by employing a spherical shape and the Hertz-Mindlin elastic model. However, this approach is unsuitable due to green pellets’ non-spherical shape and elastic–plastic properties. This study introduces the incorporation of the actual shape of green pellets and a hysteretic spring elastoplastic contact model. In this research work, a novel design of rolls with a grooved surface was recommended for the first time, resulting in a Total Screening Efficiency (TSE) of 96.4% compared to 87.1% for rolls with a smooth surface. Additionally, while exploring the impact of the decks number on the roller screen, a new design, known as the banana roller screen, was introduced. Under similar conditions, the TSE increased by 1.22% compared to the general roller screen type. It was also found that the roller screen’s capacity is directly related to the diameter, speed of rotation, and the number of rolls, while it has a reverse relationship with TSE and deck angle.