In this paper, we analyze the outage performance of a rate-splitting multiple access (RSMA)-aided semi-grant-free (SGF) transmission system, in which a grant-based user (GBU) and multiple grant-free users (GFUs) access the base station by sharing the same resource blocks. In the RSMA-aided SGF (RSMA-SGF) transmission system, the GBU and admitted GFU are respectively treated as the primary and secondary users by using the cognitive radio principle. With the aid of RSMA, the admitted GFU’s transmit power allocation, target rate allocation, and successive interference cancellation decoding order are jointly optimized to attain the maximum achievable rate for the admitted GFU, without deteriorating the GBU’s outage performance compared to orthogonal multiple access. Taking into account the extended non-outage zone achieved by rate-splitting, a closed-form expression is derived for the outage probability of the admitted GFU in the considered RSMA-SGF system. Asymptotic analysis for the admitted GFU’s outage probability is also provided. The superior outage performance and full multiuser diversity gain achieved by the RSMA-SGF transmission system are verified by the analytical and simulation results.