Following the excavation of the rectangular tunnel faces with deep harder rock, spalling often occurs on both sidewalls of the tunnel, which affects the tunnel support and stability. To investigate the spalling phenomenon of the harder rock under the deep initial three-dimensional (3D) stress conditions, true triaxial loading tests of different depths (250, 500, 750, and 1000 m) and different stress combinations (four different horizontal stresses at the same depth 1000 m) were designed on plastic fine grain marble. The overall failure process of the surrounding rock was monitored and recorded by a wireless micro-camera. The results show that, under the shallow stress condition (250 and 500 m), the V-shaped bands will appear on the both sidewall due to the overall shear failure of the specimen. Under the deep stress condition (750 and 1000 m), the obvious spalling was observed on the hole sidewalls of all marble specimens. Moreover, at the time of hole sidewall spalling, the vertical stress is the maximum principal stress. The spalling process of the hole sidewall can be divided into four stages: the microcrack initiation, the appearance of thin rock slabs, the formation of arc-shaped notches, and the expansion to V-shaped notches. If the excavation depth remains the same (1000 m) when the maximum principal stress is perpendicular to the tunnel axis, excavation along the minimum principal stress direction is beneficial to reducing the spalling damage degree. Compared to the brittle granite specimens with rectangular holes under the same 3D stress conditions, the influence of brittleness and plasticity of rock materials on spalling failure was analysed. (1) Spalling failure mode of hole sidewalls: the plastic-marble is static failure without fine particle ejection, while the brittle granite will break dynamically with fine particle ejection. (2) Spalling failure process and zones of sidewall: the cracks on the plastic marble sidewalls develop parallel to the axial direction, and run through the whole hole sidewall along the axis, while the cracks on the brittle granite hole sidewalls propagate perpendicular to the axial direction, and the sidewalls first break locally and then break through the whole hole. Failure zones on both sidewalls of the two kinds of rock specimens will transit from arc-shaped notch to V-shape notch. Those failure notches are symmetrical and finally penetrate the sidewalls. (3) Microscopic failure mode: scanning electron microscope (SEM) shows that the plastic and brittle rock materials are tensile failure during the spalling under 3D stress conditions. The plastic marble is characterized by fracture tension failure, while the brittle granite is characterized by layered tension failure. Some suggestions are given finally, under the condition that the vertical stress is the maximum principal stress, when the tunnel is excavated along the horizontal direction, the resistant ability of the tunnel sidewall in the plastic and brittle surrounding rock should be strengthened by supporting the sidewall, which can effectively reduce the degree of spalling damage.