To overcome the drawbacks of the harmony search (HS) algorithm and further enhance its effectiveness and efficiency, an improved differential HS (IDHS) is proposed to solve numerical function optimization problems. The proposed IDHS has a novel improvisation scheme that integrates DE/best/1/bin and DE/rand/1/bin from the differential evolution (DE) algorithm to enhance its local search and exploration capabilities and a new pitch adjustment rule that benefits from the best solution in the harmony memory to increase its convergence speed. With dynamically adjusted parameters, the proposed IDHS can balance exploitation and exploration throughout the search process. The numerical results of an experiment with classic testing functions and those of a comparative experiment show that IDHS outperforms eight algorithms in the HS family and three widely used population-based algorithms in different families, including DE, particle swarm optimization, and improved fruit fly optimization algorithm. IDHS demonstrates fast convergence and an especially good capability to handle difficult high-dimensional optimization problems.