In this investigation, a novel opto-mechatronics design of linear Laser encoder system which can eliminate geometrical errors of X-Y positioning stage was proposed. The eliminated geometrical errors, which are squareness error, straightness error and Abbe error, can be reduced by this Laser encoder system. Those errors are induced by the mechanical components and assemblies. In this design, a pentaprism beam splitter was employed to divide the laser beam into two encoder axes which are perpendicular to each other. For this arrangement, a real zero point of the positioning stage was set inside the pentaprism beam splitter. Therefore, those specific geometrical errors can be minimized by the proposed system. In this research, the design of the opto-mechatronics structure and its theoretical simulations will be studied. The experimental results which were obtained in the ordinary environment revealed that the resolution of the proposed positioning stage is about 15.8 nm. In addition, the maximum standard deviation of static positioning error is about 50 nm. The Laser encoder system presented in this study is recommended to be used in the precision machinery and semiconductor industries for the precision positioning purpose.