Pre-trained language model with feature reduction and no fine-tuning
Pre-trained language models were proven to achieve excellent results in Natural Language Processing tasks such as Sentiment Analysis. However, the number of sentence embeddings from the base model of Bidirectional Encoder from Transformer (BERT) is 768 for a sentence, and there will be more than mil...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Book Section |
| Published: |
Springer Science and Business Media Deutschland GmbH
2022
|
| Subjects: | |
| Online Access: | http://eprints.utm.my/id/eprint/100760/ http://dx.doi.org/10.1007/978-981-19-3923-5_59 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Summary: | Pre-trained language models were proven to achieve excellent results in Natural Language Processing tasks such as Sentiment Analysis. However, the number of sentence embeddings from the base model of Bidirectional Encoder from Transformer (BERT) is 768 for a sentence, and there will be more than millions of unique numbers when the dataset is huge, leading to the increasing complexity of the system. Thus, this paper presents the feature reduction of the sentence embeddings classification with BERT to decrease the number of features and complexity by using feature reduction algorithm. With 50% fewer features, the experimental results show that the proposed system improves the accuracy by 1%−2% with 89% lesser GPU memory usage. |
|---|