Fuzzy preference matroids rough sets for approximate guided representation in transformer

Abstract

Recently, the transformer has exhibited remarkable performance across various applications, primarily owing to its exceptional capability in capturing global information through the attention mechanism. Nevertheless, the dot-product within attention results in inaccurate and confusing guidance in the presence of uncertain features, subsequently affecting robustness. Hence, introducing an approximate guided method to handle uncertain features is essential for achieving enhanced objectivity. In this study, we employed fuzzy preference relations to construct a matroids structure, introducing it into rough sets to form the fuzzy preference matroids rough sets. Detailed mathematical properties and axiomatic proofs are presented. The closed sets within the matroids form a subspace of the lower approximation proving that the proposed satisfy the approximation ability of rough sets. Further, an approximate guided representation method with a lower approximation of the matroids has been developed. It is integrated into a plug-and-play transformer block that can be flexibly deployed across various tasks. Experimental validation has demonstrated the outstanding performance of this method across 3D point cloud classification, named entity recognition, multimodal emotion recognition, and image classification. Specifically, on the Weibo NER and the IEMOCAP datasets, attain state-of-the-art (SOTA) F1-scores. The resource code is validated at https://github.com/WinnieSunning/FPMRST.