In chiral effective field theory the leading order (LO) nucleon-nucleon potential includes two contact terms, in the two spin channels $S=0,1$, and the one-pion-exchange potential. When the pion degrees of freedom are integrated out, as in the pionless effective field theory, the LO potential includes two contact terms only. In the three-nucleon system, the pionless theory includes a three-nucleon contact term interaction at LO whereas the chiral effective theory does not. Accordingly arbitrary differences could be observed in the LO description of three- and four-nucleon binding energies. We analyze the two theories at LO and conclude that a three-nucleon contact term is necessary at this order in both theories. In turn this implies that subleading three-nucleon contact terms should be promoted to lower orders. Furthermore this analysis shows that one single low energy constant might be sufficient to explain the large values of the singlet and triplet scattering lengths.