The anisotropic thermal diffusivity of aligned carbon nanotube-polymer composites was determined using a photothermoelectric technique. The composites were obtained by infiltrating poly-dimethyl siloxane (PDMS) in aligned multiwall CNT arrays grown by chemical vapor deposition on silicon substrates. The thermal diffusivities are insensitive to temperature in the range of 180 K-300 K. The thermal diffusivity values across the alignment direction are approximately 2-4 times smaller than along the alignment direction and larger than effective media theory predictions using reported values for the thermal diffusivity of millimeter thick aligned multiwall carbon nanotube arrays. The effective room temperature thermal conductivity of the composite along the carbon nanotube alignment direction is at least 6X larger than the thermal conductivity of the polymer matrix and is in good agreement with the effective media predictions. This work indicates that infiltration of long and aligned carbon nanotube arrays is currently the most efficient method to obtain high thermal conductivity polymer composites.