Abstract The telescoping ballbar is a versatile and easy-to-use error identification instrument for machine tools. The ballbar movement trajectory used while machining is limited to acquiring a large circular arc for positioning accuracy evaluation. However, it cannot be used to calculate the expected machine tool error parameters. Herein, a helical path is explored because it provides a complete helix turn under machining conditions, similar to the general ballbar path during a non-machining test, and enables the calculation of relevant machine tool error parameters. The theoretical model and program for a helical ballbar test are developed and validated experimentally on two different research platforms: a hexapod-based machining cell and a Mitsui Seiki HU40-T five-axis machine tool. Using a Renishaw QC-20W ballbar system with a 50 mm radius, the tests were conducted with helical pitches of 0, 0.1, 0.2, 0.5, 1, and 1.5 mm, respectively. After compensation, the helical-based ballbar test showed similar radial deviation curve patterns as the standard ballbar test, with differences in circular deviation of less than 2.4 µm and 1.5 µm observed for the hexapod machining cell and machine tool, respectively. The proposed helical ballbar measurement and data processing method is therefore feasible and can produce comparable results to circular ballbar tests.