Since the data generation rate of high-resolution satellites is increasing rapidly, to relieve the stress of data downloading and processing systems while enhancing the time efficiency of information acquisition, it is important to deploy on-board edge computing on satellites. However, the volume, weight, and computability of on-board systems are strictly limited by the harsh space environment. Therefore, it is very difficult to match the computability and the requirements of diversified intelligent applications. Currently, this problem has become the first challenge of the practical deployment of on-board edge computing. To match the actual requirements of the Luojia3 satellite of Wuhan University, this manuscript proposes a three-level edge computing architecture based on a System-on-Chip (SoC) for low power consumption and expandable on-board processing. First, a transfer level is designed to focus on hardware communications and Input/Output (I/O) works while maintaining a buffer to store image data for upper levels temporarily. Second, a processing framework that contains a series of libraries and Application Programming Interfaces (APIs) is designed for the algorithms to easily build parallel processing applications. Finally, an expandable level contains multiple intelligent remote sensing applications that perform data processing efficiently using base functions, such as instant geographic locating and data picking, stream computing balance model, and heterogeneous parallel processing strategy that are provided by the architecture. It is validated by the performance improvement experiment that following this architecture, using these base functions can help the Region of Interest (ROI) system geometric correction fusion algorithm to be 257.6 times faster than the traditional method that processes scene by scene. In the stream computing balance experiment, relying on this architecture, the time-consuming algorithm ROI stabilization production can maintain stream computing balance under the condition of insufficient computability. We predict that based on this architecture, with the continuous development of device computability, the future requirements of on-board computing could be better matched.