The published article proposes economic and mathematical models for predicting the profits of an enterprise that incurs certain transaction costs in addition to production costs. The source of transaction costs of an enterprise is the compelled expenses for the search and processing of economic information, financing of negotiations, elaboration and conclusion of contracts with partners, organization of protection of property rights, payment of opportunistic behavior of employees and managers of the enterprise, etc. A numerical analysis of the obtained economic and mathematical models for predicting the economic indicators of an enterprise having a certain level of transaction costs shows that instead of the maximum level of profit, an enterprise can only achieve its lower optimal level, which corresponds to the optimal value of the transaction utility function.