In [Spectral asymmetry and Riemannian geometry. III, Math. Proc. Cambridge Philos. Soc. 79 (1976) 71–99] Atiyah, Patodi and Singer introduced spectral flow for elliptic operators on odd-dimensional compact manifolds. They argued that it could be computed from the Fredholm index of an elliptic operator on a manifold of one higher dimension. A general proof of this fact was produced by Robbin–Salamon [The spectral flow and the Maslov index, Bull. London Math. Soc. 27(1) (1995) 1–33, MR 1331677]. In [F. Gesztesy, Y. Latushkin, K. Makarov, F. Sukochev and Y. Tomilov, The index formula and the spectral shift function for relatively trace class perturbations, Adv. Math. 227(1) (2011) 319–420, MR 2782197], a start was made on extending these ideas to operators with some essential spectrum as occurs on non-compact manifolds. The new ingredient introduced there was to exploit scattering theory following the fundamental paper [A. Pushnitski, The spectral flow, the Fredholm index, and the spectral shift function, in Spectral Theory of Differential Operators, American Mathematical Society Translations: Series 2, Vol. 225 (American Mathematical Society, Providence, RI, 2008), pp. 141–155, MR 2509781]. These results do not apply to differential operators directly, only to pseudo-differential operators on manifolds, due to the restrictive assumption that spectral flow is considered between an operator and its perturbation by a relatively trace-class operator. In this paper, we extend the main results of these earlier papers to spectral flow between an operator and a perturbation satisfying a higher pth Schatten class condition for [Formula: see text], thus allowing differential operators on manifolds of any dimension [Formula: see text]. In fact our main result does not assume any ellipticity or Fredholm properties at all and proves an operator theoretic trace formula motivated by [M.-T. Benameur, A. Carey, J. Phillips, A. Rennie, F. Sukochev and K. Wojciechowski, An analytic approach to spectral flow in von Neumann algebras, in Analysis, Geometry and Topology of Elliptic Operators (World Scientific Publisher, Hackensack, NJ, 2006), pp. 297–352, MR 2246773; A. Carey, H. Grosse and J. Kaad, On a spectral flow formula for the homological index, Adv. Math. 289 (2016) 1106–1156, MR 3439708]. We illustrate our results using Dirac type operators on [Formula: see text] for arbitrary [Formula: see text] (see Sec. 8). In this setting Theorem 6.4 substantially extends Theorem 3.5 of [A. Carey, F. Gesztesy, H. Grosse, G. Levitina, D. Potapov, F. Sukochev and D. Zanin, Trace formulas for a class of non-Fredholm operators: a review, Rev. Math. Phys. 28(10) (2016) 1630002, MR 3572626], where the case d = 1 was treated.