The increasing variety of products poses a challenge for efficient manufacturing on production lines due to resulting small batch sizes and frequent product changes, which lower the average overall effectiveness. Especially for industries such as the Fast Moving Consumer Goods (FMCG) industry that manufacture at high speed on production lines, it is mandatory to increase the performance of production lines in an economical way. Due to the complexity of such production lines, identifying efficient actions and combining them into economic improvement trajectories is challenging. There are numerous approaches based on different concepts, such as simulation-based heuristics, to solve these challenges. In other application areas, reinforcement learning has shown remarkable success in recent years, and first reinforcement learning-based approaches to this specific problem can be found. However, these approaches mostly focus on details instead of providing a holistic view of the possibilities to improve a production line or are limited in their practical application due to lack of integration of existing expert knowledge or limited quality of results. For this reason, this paper proposes a hierarchical reinforcement learning approach that combines discrete event simulation with a heuristically driven multi-agent system. Thus, the selection of the improvement strategy of the production line is performed by one agent, and the dedicated improvement of specific parameters is performed by specialized subordinate other agents. Through this hierarchical multi-agent system, on the one hand, the learning rate can be increased. On the other hand, by guiding the agents through a heuristic based on expert knowledge, the learning quality is increased.