The combination of full-duplex communication (FDC) and millimeter wave integrated access and backhaul (mmWave IAB) is a highly implementable technique in beyond 5G era. However, the technical challenges such as sub-channel allocation and interference coordination must be effectively handled before the attractive benefits of the commercial implementation are enjoyed. In particular, the multiple user interference (MUI) is caused when the same sub-channel is shared by the concurrent links sourced from different transmitters, and the residual self-interference (RSI) is resulted from self-interference cancellation (SIC) in FDC. To this end, a coalitional game based sub-channel allocation mechanism is proposed for the full-duplex-enabled mmWave IAB network to maximize the total actual achievable rate. Firstly, the maximization problem of the sum actual achievable rate is formulated as an integer non-convex and non-linear optimization problem under the constriction of MUI and RSI, which is a non-polynomial hard (NP-Hard) problem, and it is intractable to obtain the optimal solution. Secondly, the formulated problem is transformed as a cooperative game, and a coalitional game based sub-channel allocation (CGSA) algorithm with lower computational complexity is proposed to obtain the sub-optimal solution. Thirdly, the properties of the proposed CGSA algorithm are discussed and analyzed from the aspects of stability and convergence, and it is proved that the final coalition partition formed by the proposed CGSA algorithm is Nash-stable. Fourthly, the proposed CGSA algorithm is compared with the reference algorithms under different scenarios, extensive simulations show that superiorities of the proposed CGSA algorithm are obvious including sum of achievable rate, number-ratio of user equipments with satisfied quality-of-service (QoS)requirement and iterations of algorithm convergence.