StyleBERT: Chinese pretraining by font style information (2202.09955v2)

Abstract: With the success of down streaming task using English pre-trained LLM, the pre-trained Chinese LLM is also necessary to get a better performance of Chinese NLP task. Unlike the English language, Chinese has its special characters such as glyph information. So in this article, we propose the Chinese pre-trained LLM StyleBERT which incorporate the following embedding information to enhance the savvy of LLM, such as word, pinyin, five stroke and chaizi. The experiments show that the model achieves well performances on a wide range of Chinese NLP tasks.