The propagation characteristics of fermionic and bosonic quasiparticles determine the fundamental transport properties of solids and are of great technological relevance for designing logic devices. In particular, nonreciprocity, which describes that a quasiparticle flows preferably along a certain direction of a symmetry path, is an essential requirement to realize logic architectures, e.g., switches, diodes, transistors, etc. Here we introduce a mechanism, which leads to giant nonreciprocity of ultrafast terahertz magnons in ferromagnetic films with a large spin-orbit coupling. The mechanism is based on the competition between the exchange and spin-orbit scattering. We anticipate that the effect can be used to excite nonreciprocal or even unidirectional magnons in a large class of ultrathin films and nanostructures grown on substrates with a large spin-orbit coupling.