This paper presents a FPGA-based hardware architecture for capturing real-time panoramic images and the parallel implementation of an image stitching algorithm based on the scale-invariant feature transform (SIFT) features. First, the system transfers the images captured by five CMOS sensors to the designed FPGA board through the FMC adapter plate. Then, the system employs the FPGA-based parallel computing to extract the SIFT features and detect the best matching region and points for image registration. Finally, in order to obtain the panoramic image with smooth changes at the junctions, images of the adjacent images are fused. Using the designed FPGA platform makes it possible to extract the SIFT and stitch the images in real-time, which is not realized in most panoramic imaging systems. The panoramic system, developed using the Xilinx Zynq®-7000 all programmable SOC, can capture video of 5*640*480 images at the speed of 60 fps. To enable high-speed image transfer from the imaging system to the host computer, the USB3.0 interface with the transmission speed of 359MB/s has been developed. The real-time efficiency of the parallel system will be demonstrated by the simulations and experiments.