In this note, we prove local-in-time well-posedness for a fully dispersive Boussinesq system arising in the context of free surface water waves in two and three spatial dimensions. Those systems can be seen as a weak nonlocal dispersive perturbation of the shallow-water system. Our method of proof relies on energy estimates and a compactness argument. However, due to the lack of symmetry of the nonlinear part, those traditional methods have to be supplemented with the use of a modified energy in order to close the a priori estimates.