Data mining is recognized as highly sought-after service. Many universities are involved in training specialists in this field. However, consumer demands are increasing, resulting in higher requirements for the mathematical models used in this regard. Clients need to be sure in the quality level of the recommendations they get. One method to achieve this is provided with the use of Explainable Artificial Intelligence (XAI) technology. The current situation worldwide poses limitations on the use of foreign software, and domestic analytical systems available to universities may not have XAI module among their functions. The solution is to develop such system independently. The authors developed a software product that enables training of three types of mathematical models, each capable of generating an explanation in textual format for its forecasts. A series of experiments was conducted in order to confirm functionality of all elements of the created product, as well as its potential to be used training processes in higher education system.