The shortage of natural aggregates has compelled the developers to devote their efforts to finding alternative aggregates. On the other hand, demolition waste from old constructions creates huge land acquisition problems and environmental pollution. Both these problems can be solved by recycling waste materials. The current study aims to use recycled brick aggregates (RBA) to develop eco-friendly pervious concrete (PC) and investigate the new concrete’s structural performance and pore structure distributions. Through laboratory testing and image processing techniques, the effects of replacement ratio (0%, 20%, 40%, 60%, 80%, and 100%) and particle size (4.75 mm, 9.5 mm, and 12.5 mm) on both structural performance and pore feature were analyzed. The obtained results showed that the smallest aggregate size (size = 4.75 mm) provides the best strength compared to the large sizes. The image analysis method has shown the average pore sizes of PC mixes made with smaller aggregates (size = 4.75 mm) as 1.8–2 mm, whereas the mixes prepared with an aggregate size of 9.5 mm and 12.5 mm can provide pore sizes of 2.9–3.1 mm and 3.7–4.2 mm, respectively. In summary, the results confirmed that 40–60% of the natural aggregates could be replaced with RBA without influencing both strength and pore features.