Under uniaxial loading, the heterogeneity of rock mass, the change of mechanical properties of joints, and the progressive fracture process of rock mass significantly affect the size effect and anisotropy of columnar jointed basalts (CJBs). To reveal the fracture features and failure mechanisms of CJBs influenced by the size effect and anisotropy, the digital image correlation (DIC), meso-damage mechanics, statistical strength theory, and continuum mechanics were combined (the DIC-improved RFPA), the digital images of CJB specimens were processed to establish the inhomogeneous numerical models, and a series of numerical tests were therefore conducted. The gradual fracture processes and macro failure patterns of CJBs orthogonal and parallel to column axis under uniaxial compression were studied, and the effects of various factors on the size effect and anisotropic mechanical properties of CJBs were further analyzed. The results show that model size, column dip angle, rock heterogeneity, column diameter, elastic modulus of joints, residual strength coefficient of joints, ratio of shift distance of joints, irregularity degrees of columns, and model boundaries have remarkable and complex effects on the mechanical behaviors of CJBs. The results greatly improve our understanding of the non-linear deformation and failure behaviors of CJBs and provide theorical basis for engineering construction in the areas of CJBs.