As systems for utilizing geothermal energy, energy piles have been widely employed in many engineering applications, and they offer various advantages. A significant amount of research has been devoted toward developing methods to enhance the heat exchange efficiency of these systems, including studies on the shape of the heat exchange tube and the shape and spacing of the energy piles. After around four decades of research and development, existing methods for improving the heat exchange efficiency have become relatively mature, thus hindering further improvements. Therefore, it is essential to develop an alternative approach for enhancing the heat transfer and improving the heat exchange efficiency of these systems. To this end, this paper first summarizes the conventional heat transfer model and then elaborates on the selection of the concrete pile material as the heat transfer medium. In this study, samples of graphite concrete and silicon carbide (SiC) concrete were prepared. Subsequently, several tests on the heat transfer behaviors and mechanical performances of the samples were conducted, in order to determine the physical and mechanical properties of the modified concrete. The results showed that, at the same ambient temperature, when the graphite content exceeded 15%, the heat transfer coefficient increased, leading to a significant improvement in heat transfer. However, the compressive strength decreased noticeably. With an increase in the SiC content, the thermal conductivity of the concrete gradually increased, while the compressive strength of the concrete with SiC increased considerably. The results proved that modifying the concrete used in energy piles is a feasible method for improving heat transfer.