We perform a theoretical investigation of the instability of a helical vortex filament beneath a free surface in a semi-infinite ideal fluid. The focus is on the leading-order free-surface boundary effect upon the equilibrium form and instability of the vortex. This effect is characterised by the Froude number $F_r = U(gh^*)^{-{1}/{2}}$ where $g$ is gravity, and $U = \varGamma /(2{\rm \pi} b^*)$ with $\varGamma$ being the strength, $2{\rm \pi} b^*$ the pitch and $h^*$ the centre submergence of the helical vortex. In the case of $F_r \rightarrow 0$ corresponding to the presence of a rigid boundary, a new approximate equilibrium form is found if the vortex possesses a non-zero rotational velocity. Compared with the infinite fluid case (Widnall, J. Fluid Mech., vol. 54, no. 4, 1972, pp. 641–663), the vortex is destabilised (or stabilised) to relatively short- (or long-)wavelength sub-harmonic perturbations, but remains stable to super-harmonic perturbations. The wall-boundary effect becomes stronger for smaller helix angle and could dominate over the self-induced flow effect depending on the submergence. In the case of $F_r > 0$ , we obtain the surface wave solution induced by the vortex in the context of linearised potential-flow theory. The wave elevation is unbounded when the $m$ th wave mode becomes resonant as $F_r$ approaches the critical Froude numbers ${\mathcal {F}} (m) = (C_0^*/U)^{-1} (mh^*/b^*)^{-{1}/{2}}$ , $m=1, 2, \ldots,$ where $C_0^*$ is the induced wave speed. We find that the new approximate equilibrium of the vortex exists if and only if $F_r < {\mathcal {F}}(1)$ . Compared with the infinite fluid and $F_r \rightarrow 0$ cases, the wave effect causes the vortex to be destabilised to super-harmonic and long-wavelength sub-harmonic perturbations with generally faster growth rate for greater $F_r$ and smaller helix angle.