We develop a semiclassical theory of wave propagation based on invariant Lagrangian manifolds existing in conservative Hamiltonian systems with chaotic dynamics. They are stable and unstable manifolds of unstable periodic orbits, and their intersections consist of homoclinic and heteroclinic orbits. For arbitrarily long times, we find matrix elements of the evolution operator between wave functions constructed in the neighbourhood of short unstable periodic orbits, in terms of canonical invariants of homoclinic and heteroclinic orbits. We verify the accuracy of these expressions by computing millions of homoclinic orbits and thousands of heteroclinic ones in the hyperbola billiard. This second part in which we describe off-diagonal matrix elements serves as complement to a previous article (first part) where we have described diagonal matrix elements.