We present a fourth order accurate finite difference method for the elastic wave equation in second order formulation, where the fourth order accuracy holds in both space and time. The key ingredient of the method is a boundary modified fourth order accurate discretization of the second derivative with variable coefficient, (μ(x)u x ) x . This discretization satisfies a summation by parts identity that guarantees stability of the scheme. The boundary conditions are enforced through ghost points, thereby avoiding projections or penalty terms, which often are used with previous summation by parts operators. The temporal discretization is obtained by an explicit modified equation method. Numerical examples with free surface boundary conditions show that the scheme is stable for CFL-numbers up to 1.3, and demonstrate a significant improvement in efficiency over the second order accurate method. The new discretization of (μ(x)u x ) x has general applicability, and will enable stable fourth order accurate approximations of other partial differential equations as well as the elastic wave equation.