Enhancing separation and reaction in solar membrane reactors is critical for achieving temperature reduction and efficiency improvement. Based on the topology optimization method, this study investigates the optimal catalyst bed porosity distribution in solar methane steam reforming membrane reactor to maximize separation and reaction, while revealing the performance enhancement mechanism. The final optimized distribution exhibits a uniform high porosity distribution in the front and numerous inclined radial paths in the middle and rear, which achieves enhanced radial bidirectional transport. This bidirectional transport enhancement is the intrinsic mechanism for the synergistic effect of reactions and separations in reactor, thus enhancing comprehensive performances. The optimized porosity distributions with various original reaction levels and reactor lengths tend to be similar in topology optimization. They always improve solar membrane reactor performances, especially at lower reaction levels and longer reactors. At the original methane conversion of 0.6, the optimized solar membrane reactor has the maximum improvement with 28.9%, 42.8% and 32.0% in methane conversion, fuel efficiency and hydrogen recovery, respectively, which reaches 92.3% of the ideal performance. Moreover, higher relative improvements in overall performance of 30.9–59.0% are reached at reactor length of 3 m.