Unsupervised Domain Adaptation for Sentimental Classification by Word Embeddings on the Lower Layer of BERT

Abstract

Bidirectional Encoder Representations from Transformers (BERT) is a stacked (12 or 24) model of multi-head attention applied in the transformers. The multi-head attention of each layer outputs a word embedded expression sequence corresponding to the input word sequence. When BERT is applied to the feature base, the output is the word-embedded expression column of the highest layer used in each task. On the other hand, in domain adaptation, projecting the data of each region onto the common subspace of the source and target domains is an effective approach. When constructing a feature vector on a common subspace from a word-embedded representation output of the BERT, the most significant layer depends on the task learning task assigned to BERT, so is not necessarily more significant than the word-embedded representation of a lower layer. Layers are suboptimal for regional adaptation. Here we confirm this concept on unsupervised domain adaptation of an emotion analysis.