The concept of invariance relates to both the intrinsic symmetries of physical systems and the symmetry of the set of equivalent reference frames used to observe them. Standard algebraic expressions for electrostatic potentials and crystal-field effective operators display both types of invariance. The concept of a reference frame is generalized to that of an ‘observing system’, which can, for example, be the basis states of a quantum system. This idea is related to Racah’s mathematical machinery for evaluating the matrix elements of many-electron 4f open-shell states in lanthanide ions. It is argued, on the basis of computational flexibility and ease of interpretation, that all equations that represent physical processes be expressible in terms of invariants of the set of observing systems. This ‘Principle of Invariance’ is then applied to special relativity, leading to a simple geometrical interpretation of Maxwell’s electromagnetic field equations. The close relationship between Dirac’s relativistic wave equation and Maxwell’s equations is then exposed. This leads to the concept of an inner structure of space-time and the reinterpretation of particle spin. Finally, it is shown that the use of invariants in relativity theory identifies a set of observing systems with a higher symmetry than that of Minkowski space-time.