Shear fractures are common in the rock mass, especially with increasing depth, due to the increase in lateral pressure. Hence, investigating these fractures is crucial for improving the stability and safety of underground structures at greater depths. The Short Core in Compression (SCC) method has recently been used by some researchers to generate shear fractures under mode II loading and initiation due to the ease of sample preparation and lateral pressure application. This study examines the impact of sample geometry and lateral pressure on the formation of shear fractures in rocks, by conducting numerical and experimental investigations on SCC samples. In this paper, the effect of sample shape, notch length, and lateral pressure was evaluated depending on the shear and the maximum principal stress distribution. To the best knowledge of the authors, the impact of these factors has not been sufficiently explored in the literature. Moreover, the mode II stress intensity factor, fracture toughness, and fracture path were investigated. The results provided the stress intensity factor for different geometries of the SCC sample that can be used for many applications in mechanical and geological engineering. Our findings showed that the resulting fracture must be evaluated due to the possibility of crack growth under shear σxy or maximum principal stress σmax, especially for Notch Length/Diameter ≥0.5. Additionally, lateral pressure mainly affects the distribution and magnitude of σmax, such that σmax is inversely proportional to this pressure. The results showed that the SCC sample with Height/Diameter = 2 is preferred because σmax is minimum for this sample shape, which promoted fracture growth under shear stress not tensile stress. Furthermore, as the lateral pressure increases, mode II fracture toughness significantly increases and the number of fragments that spalled off the specimen along the fracture path decreases.