Binary and ternary transition metal borides exhibit many outstanding properties, such as high melting temperature, high temperature strength, and electrical and thermal conductivities. However, defect properties have been rarely studied so far. Here, using first-principles calculations, we investigate structural effects on defect recovery processes in MB and MAB phases focusing on how Al layers and the type of transition metal affect annihilation of defects. First, we find that metal interstitials cannot be accommodated in MAB phases that have two Al layers, but instead they form antisites, which are largely immobile and therefore detrimental for defect recovery processes. We also find that the absence of Al layers in MB phases that have B chains leads to faster migration of metal interstitials and lower recombination barriers of B Frenkel pairs and hence contributes to efficient defect recovery processes of those MB phases as compared to the corresponding MAB phases. In the other types of MB and MAB phases that have B rings, we find that the role of Al layers varies depending on the type of metal elements. Lastly, we find that a larger M–B bond strength, which depends on the type of transition metal in MAB phases, correlates with a higher recombination barrier of B Frenkel pairs. Based on the discovered trends, we provide insights into the selection of materials to help design diverse applications where defect recovery processes are important.