Filled rubber is modelled as a volume filled with hard spherical inclusions (filler) connected to each other via hyperelastic links (polymer in gaps). These inclusions are arranged into a network of fractal clusters, which are observed in real filled rubbers. The force response of links to external loads is calculated by the finite element method applied to solve the problem of pair interaction of inclusions in the polymer matrix. The obtained dependencies are used then in a structural–mechanical model of the deformation of a representative volume of inclusions. At a certain elongation of the link the volume is broken and in consequence the Mullins softening effect is modeled at the macroscopic level. Therefore, the model explicitly takes into account the specific features of the microstructure and allows one to establish a direct relation between the micro and macro properties of the material. In addition to the clustered structure, the stress–strain behavior of the material with periodic and random arrangements of inclusions (filler volume fraction: 0.13 and 0.20) is simulated. It is shown that the structure with a regular arrangement of filler particles is less resistant to large deformations than the structure in which filler particles are arranged in a disordered manner. It was found that during deformation the initially close inclusions of clustered or random structures move in groups, maintaining their relative position. The largest local deformations of material in the gaps occur between inclusions where the initial gap is at least equal to size of inclusion.