It remains a challenge for discrete element method to simulate large-scale systems with arbitrarily shaped particles. This paper presents a novel approach called Ray Tracing Discrete Element Method (RTDEM) to offer ultra-high efficiency in simulating particles with arbitrary shapes. In RTDEM, we use a triangular mesh to represent the surface of a particle for greater flexibility in shape representation. Template meshes are further introduced to group particles of different shapes to save computing memory. Key to RTDEM are innovative ray tracing-based algorithms specially designed for accurate and efficient contact detection and resolution between arbitrarily shaped particles. Based on the RT algorithms, discrete potential field functions are further proposed for the discrete mesh vertices and facets to define energy-conservative contact models that include vertex-potential, facet-potential models, and their combination. The integration of these key elements into RTDEM offers a whole new DEM framework that readily empowers any parallel computing architecture for high acceleration on granular media modeling. It matches particularly well with the Graphics Processing Units (GPUs) to fully unleash the computing power of ray tracing cores for large-scale DEM simulations of arbitrarily shaped particles, even on a personal computer with a common gaming GPU card.