The noise and high randomness of the stock market are primary obstacles to profitability. These factors cause stock returns to consist of short-term predictable and stock-specific residual parts. Therefore, it is beneficial to separate and model the two items for forecasting returns in the randomness environment. On this basis, we propose a novel Temporal Return Separation and Restore Forecasting (TRSRF) method. TRSRF first extracts price- and relation-based features from historical stock inputs and uses them to decompose and restore predictable and residual elements of historical returns , respectively. Then it forecasts these components based on historical values. We conducted experiments on the Chinese small and mid-cap, blue-chip, and large-cap indexes over ten years to compare with seven proposed baselines. On the three Chinese indexes our method separately exceeds 6%, 16%, and 22% absolute annualized returns than our baseline. The ablation study also proves the necessity of return decomposition and inter-stock relation modeling. We find that modeling inter-stock relations on small-cap stocks works better than on large-cap stocks. Furthermore, we prove our method might be an implicit ensemble model of the predictable and residual parts, which is why it works stability under various market conditions.