We present a 1-loop toroidal membrane winding sum reproducing the conjectured $M$-theory, four-graviton, eight derivative, $R^4$ amplitude. The $U$-duality and toroidal membrane world-volume modular groups appear as a Howe dual pair in a larger, exceptional, group. A detailed analysis is carried out for $M$-theory compactified on a 3-torus, where the target-space $Sl(3,\Zint)\times Sl(2,\Zint)$ $U$-duality and $Sl(3,\Zint)$ world-volume modular groups are embedded in $E_{6(6)}(\Zint)$. Unlike previous semi-classical expansions, $U$-duality is built in manifestly and realized at the quantum level thanks to Fourier invariance of cubic characters. In addition to winding modes, a pair of new discrete, flux-like, quantum numbers are necessary to ensure invariance under the larger group. The action for these modes is of Born-Infeld type, interpolating between standard Polyakov and Nambu-Goto membrane actions. After integration over the membrane moduli, we recover the known $R^4$ amplitude, including membrane instantons. Divergences are disposed of by trading the non-compact volume integration for a compact integral over the two variables conjugate to the fluxes -- a constant term computation in mathematical parlance. As byproducts, we suggest that, in line with membrane/fivebrane duality, the $E_6$ theta series also describes five-branes wrapped on $T^6$ in a manifestly U-duality invariant way. In addition we uncover a new action of $E_6$ on ten dimensional pure spinors, which may have implications for ten dimensional super Yang--Mills theory. An extensive review of $Sl(3)$ automorphic forms is included in an Appendix.