This work presents a comparative study on clay bricks made with a local soil (Midar, eastern Morocco) incorporating with olive pomace (OP). The effect of olive pomace on the engineering properties of clay bricks before and after baking process was investigated. The main objective is to show the effectiveness of earth bricks bio-based with (OP) in improving the thermal insulation of buildings. The first step was to carry out geotechnical analysis of the soil. Different percentage addition ratios: 0, 5, 10, 15, and 20% in weight of OP were adopted in order to evaluate the thermal impact of olive pomace on fired and unfired clay bricks. The thermal transport properties of prepared specimens were measured using the hot disc method. The particle size distribution showed a clay fraction of 45.2% for the soil. The Atterberg limits show a moderate plasticity for this soil. The X-ray diffraction analysis of the studied soil shows that it is composed of several minerals (Kaolinite, Illite, Quartz, Calcite and Goethite). Thermal analysis by the Hot Disk method showed a reduction of (43.1–39.7%) in thermal conductivity for the clay-olive pomace 20% (dried-baked) composites respectively, compared to earth alone, resulting in a significant increase in thermal resistance. Finally, a comparative analysis at a building-scale using TRNSYS was carried out. Two scenarios were considered: building with (i) conventional fired clay brick walls and (ii) OP-fired clay brick walls. The results respectively show a reduction of 23.7% and 6.3% in the annual heating and cooling loads, confirming the efficiency of the clay-olive pomace composites.