Magnetoactive elastomers (MAEs) exhibit magneto-mechanical coupling and typically contain micron-sized spherical ferromagnetic particles embedded in an elastomeric matrix. Anisotropic particle arrangement in MAEs, achieved by curing the material in a magnetic field, offers tailored and enhanced magneto-mechanical coupling compared to isotropic configurations. While tunable MAE properties such as the effective modulus and vibrational response have been relatively well studied, a thorough understanding of their fracture mechanisms, particularly in MAEs with microstructural anisotropy or composites with similar structures remains almost completely unexplored. Here, we characterize the fracture mechanisms of anisotropic and isotropic unmagnetized MAEs using experiments and simulations, focusing on the effects of spherical particles in anisotropic chain-like configurations. Specifically, we experimentally measure the fracture toughness of MAEs containing different volume fractions of particles, and with particles arranged both randomly and aligned at 0°, 45°, and 90° to the loading direction. Results show that anisotropic MAEs with particle chains aligned with the load direction can improve fracture toughness by up to almost 600%, whereas isotropic MAEs increase the fracture toughness by only 420%, compared to the pure elastomer. Scanning electron microscopy of the post-fractured surface reveals toughening mechanisms at micro-scale, such as chain waviness, particle agglomeration, and particle distribution. Simulations using a finite element method-based decoupled phase field-cohesive zone model on simplified geometries qualitatively support imaging interpretations. Overall, this work explains how internal geometry, including waviness in particle chains, chain alignment, and particle agglomerations affects fracture of MAEs and generally soft composites with chain-like geometries of spherical particles. These results have engineering applications in improving fracture properties of smart soft composites relevant to soft robotics, tunable vibration absorbers, and noise attenuators.