Relay satellite network has become a promising architecture to provide rapid-response data offloading services for delay-sensitive space data. However, stochastic resources and dynamic inter-satellite contacts bring plenty of difficulties under delay constraints, which affects the transmission efficiency directly. In this paper, we first systematically analyze the data transmission process with delay guarantees and formulate a long-term stochastic optimization problem without a priori statistical knowledge. To tackle this problem, we first transform it into several sub-problems using a queue-stability-related technology. Then, we propose a transmission optimization with delay guarantees algorithm to maximize the system utility by optimizing resource allocation and channel assignment. Furthermore, we provide performance analysis to prove that the proposed algorithm can achieve long-term stability and approach near-optimal system utility. Finally, extensive simulations demonstrate the effectiveness of the proposed algorithm for system utility maximization and delay control.