Fine-grained personalized federated learning via transformer in the transformer framework

Abstract

Federated learning (FL) stands out as one of the most decentralized and privacy-preserving machine learning techniques. FL enables collaborative model training across multiple clients while preserving the privacy of their data. An extension of this method, namely, personalized federated learning (PFL), seeks to address client data heterogeneity by tailoring individualized models. Existing PFL methods that use transformers do not always take into account how federated averaging algorithms (a common FL algorithm that takes a weighted average of model parameters) can aggregate the self-attention mechanism. This makes it harder for transformers to work in federated settings. In addition, these approaches typically underperform on fine-grained tasks. To overcome these limitations, this study develops a novel transformer-in-transformer facilitated FL framework that not only aggregates global model parameters but also pinpoints personalized parameters for each client. This framework departs from traditional methods by implementing an “adaptive-personalization” mechanism, which not only enhances inter-client cooperation but also improves the model's scalability and adaptability. On the server side, a hypernetwork is developed to generate personalized projection matrices for use in the self-attention layers. These matrices determine the query, key, and value parameters for each client. Experimental results demonstrate that the model equipped with this adaptive-personalization mechanism considerably outperforms traditional models in scenarios with heterogeneous data, thus validating its effectiveness and efficiency.