We present a complete numerical model of the afterglow of a laterally structured relativistic ejecta from the radio to very high energies (VHE). This includes a self-consistent calculation of the synchrotron radiation, with its maximum frequency, and of synchrotron self-Compton (SSC) scattering that takes the Klein-Nishina regime into account. Attenuation due to pair production is also included. This model is computationally efficient and allows multi-wavelength data fitting. As a validation test, the radiative model was used to fit the broad-band spectrum of GRB 190114C at 90 s up to the TeV range. The full model was then used to fit the afterglow of GW 170817 and predict its VHE emission. We find that the SSC flux at the peak was much dimmer than the upper limit from H.E.S.S. observations. However, we show that either a smaller viewing angle or a higher external density would make similar off-axis events detectable in the future at VHE, even above 100 Mpc with the sensitivity of the Cherenkov telescope array. High external densities are expected in the case of fast mergers, but the existence of a formation channel for these binary neutron stars is still uncertain. We highlight that VHE afterglow detections would help to efficiently probe systems like this.