Caches are vulnerable to side-channel attacks as a type of shared hardware resource. Most existing defense methods can be categorized into two categories: partition and randomization. However, cache partition will bring a large performance overhead and randomization has been proven to be not safe enough. In this paper, we propose a design called ExtendCache. Different from other defense schemes applied to the entire cache, ExtendCache focuses on cache sets that may cause information leakage, which can greatly reduce the impact on performance. Based on the fact that a conflict-based attack requires a large number of accesses to the target set, ExtendCache locates the cache set that may contain sensitive information and be stolen by the attacker according to the number of accesses. After locating a suspicious set, it extends the set by borrowing cache lines from another set. This prevents attackers from controlling the state of the target set and observing effective victim behavior. We implemented ExtendCache on gem5 and took the modular square algorithm as an example to prove the effectiveness of our design. When evaluated using the SPEC2017 benchmarks, ExtendCache had minimal impact on performance, with only a 0.48% average performance loss observed.