General Words Representation Method for Modern Language Model

• Main Authors: Abbas Saliimi, Lokman, Mohamed Ariff, Ameedeen, Ngahzaifa, Ab. Ghani
• Format: Article
• Language: English
• Published: UTeM 2023
• Subjects: QA75 Electronic computers. Computer science; QA76 Computer software
• Online Access: http://umpir.ump.edu.my/id/eprint/38294/1/6241-Article%20Text-18137-1-10-20230324.pdf; http://umpir.ump.edu.my/id/eprint/38294/; https://jtec.utem.edu.my/jtec/article/view/6241

This paper proposes a new word representation method emphasizes general words over specific words. The main motivation for developing this method is to address the weighting bias in modern Language Models (LMs). Based on the Transformer architecture, contemporary LMs tend to naturally emphasize spec...