Waves running up and down the beach (‘swash’) at the landward edge of the ocean can cause changes to the beach topology, can erode dunes, and can result in inland flooding. Despite the importance of swash, field observations are difficult to obtain in the thin, bubbly, and potentially sediment laden fluid layers. Here, swash excursions along an Atlantic Ocean beach are estimated with a new framework, V-BeachNet, that uses a fully convolutional network to distinguish between sand and the moving edge of the wave in rapid sequences of images. V-BeachNet is trained with 16 randomly selected and manually segmented images of the swash zone, and is used to estimate swash excursions along 200 m of the shoreline by automatically segmenting four 1-h sequences of images that span a range of incident wave conditions. Data from a scanning lidar system are used to validate the swash estimates along a cross-shore transect within the camera field of view. V-BeachNet estimates of swash spectra, significant wave heights, and wave-driven setup (increases in the mean water level) agree with those estimated from the lidar data.