The fracture mechanism and macro-properties of SVSAC were studied using a novel test system combined with numerical simulations, which included three-point bending beam tests, the digital image correlation (DIC) technique, scanning electron microscopy (SEM), and ABAQUS analyses. In total, 9 groups and 36 specimens were fabricated by considering two critical parameters: initial notch-to-depth ratios (a0/h) and concrete mix components (seawater and volcanic scoria coarse aggregate (VSCA)). Changes in fracture parameters, such as the load-crack mouth opening displacement curve (P-CMOD), load-crack tip opening displacement curve (P-CTOD), and fracture energy (Gf), were obtained. The typical double-K fracture parameters (i.e., initial fracture toughness (KICini) and unstable fracture toughness (KICun)) and tension-softening (σ-CTOD) curve were analyzed. The test results showed that the initial cracking load (Pini), Gf, and characteristic length (Lch) of the SVSAC increased with decreasing a0/h. Compared with the ordinary concrete (OC) specimen, the P-CMOD and P-CTOD curves of the specimen changed after using seawater and VSCA. The evolution of the crack propagation length was obtained through the DIC technique, indicating cracks appeared earlier and the fracture properties of specimen decreased after using VSCA. Generally, the KICun and KICini of SVSAC were 36.17% and 8.55% lower than those of the OC specimen, respectively, whereas the effects of a0/h were negligible. The reductions in Pini, Gf, and Lch of the specimen using VSCA were 10.94%, 32.66%, and 60.39%, respectively; however, seawater efficiently decreased the negative effect of VSCA on the fracture before the cracking width approached 0.1 mm. Furthermore, the effects of specimen characteristics on the fracture mechanism were also studied through numerical simulations, indicating the size of the beam changed the fracture toughness. Finally, theoretical models of the double-K fracture toughness and the σ-CTOD relations were proposed, which could prompt their application in marine structures.