Pathology in dynamical systems I: General theory

Abstract

In Hamiltonian systems the existence of nondegenerate heteroclinic orbits connecting distinct hyperbolic periodic orbits leads to the Smale horseshoe map and symbolic dynamics [lo]. Th e complex behavior of nearby orbits implies the nonexistence of second analytic integrals for the flow [lo]. Unfortunately, proving the transversal intersection of the stable and unstable manifolds of the periodic orbits, i.e., the nondegeneracy of the heteroclinic orbits, is nontrivial in most problems. This paper presents definitions of “transversal” and “nondegenerate” which are more readily verified in applications, and which still lead to topological results on the pathology of orbits analogous to those obtained in [13]. In Part II we will give examples of Hamiltonian systems in which our definitions apply; in particular the H&on-Heiles potential [8] will be examined. Part I is done completely without smoothness assumptions. The hyperbolic periodic orbits are replaced by invariant sets which admit surfaces of section, and which are “isolated” in the sense that they are the maximal invariant sets within some neighborhood of themselves. This work generalizes several of the results of [12], and is similar in spirit to [5].