The pointing jitter originated from the random mechanical vibration of the optical platform and the noise associated with the optomechanical sensor system unquestionably deteriorate the performance of the inter-satellite optical link. The impact of the jitter changes significantly with the receiver architecture. In this paper, we present a mathematical model to investigate the link performance in the presence of angular pointing jitters for different receiver architectures. Alongside the statistical pointing error model, the derived model incorporates key receiver design parameters such as detector radius, receiver aperture size, f-number of the lens system, and beam compression ratio to study the impact of receiver architecture on the pointing jitter. As an example, a CubeSat optical receiver is analyzed. We show that by careful selection of beam compression ratio and f-number, more than 5 orders of magnitude bit error rate improvement is achievable even at large pointing error.