Improving Attentive Sequence-to-Sequence Generative-Based Chatbot Model Using Deep Neural Network Approach

• Main Author: Wan Solehah, Wan Ahmad
• Format: Thesis
• Language: English; English; English
• Published: UNIMAS 2022
• Subjects: T Technology (General)
• Online Access: http://ir.unimas.my/id/eprint/43212/6/Final%20Submission%20of%20Thesis%20Form%20-%20Signed.pdf; http://ir.unimas.my/id/eprint/43212/4/MSc%20Thesis_Wan%20Solehah%20Wan%20Ahmad%20%2824pgs%29.pdf; http://ir.unimas.my/id/eprint/43212/8/Wan%20Solehah%20ft.pdf; http://ir.unimas.my/id/eprint/43212/

Deep Neural Network (DNN) is a combination method between two different subfields of Machine Learning application, including the Artificial Neural Network (ANN) and Deep Learning (DL). An example of the DNN model is the Attentive Sequence-to-Sequence (Seq2Seq) model that was first created to tackle a problem setting in language processing. One of the applications is the chatbot model that works explicitly to accurately respond to users' inquiries. Through the years, a chatbot application has seen some improvement, from generating hard-generic responses to more flexible response. The adoption of DNN method into chatbot application produces a new generation chatbot that called as Generative-Based Chatbot. However, it is difficult to create and train a Generative-Based chatbot model that can maintain relevancy of dialogue generation in a long conversation. Hence, this research’s objective aimed to propose an optimization strategy based on Structural Modification and Optimizing Training Network for improving the lacking of accuracy of response in the chatbot application, to propose the algorithm enhancement to improve the current attention mechanism in the Attentive Sequence-to-Sequence model and the network’s training optimization of its inability to memorize the dialogue history, and lastly, to evaluate the accuracy of response of the proposed solution through data training on loss function and real data testing. The structural modification that is based on a slight modification in Additive Attention mechanism. The method is by adding a scaling factor for the dimension of the decoder hidden state. The other one is the network training’s environment optimization that is done through hyperparameter optimization by selecting and fine-tuning high impact parameters which include Optimizer, Learning Rate and Dropout to reduce error rate (loss function). The strategies applied showed that the final accuracy obtained through the training after implementing a modification in the algorithm is at 81% accuracy rate compared to the basic model that recorded its final accuracy at 79% accuracy rate. Meanwhile, after modification and optimization, the model's performance recorded its final value of accuracy and loss rate at 87% and 0.51, respectively. The result indicates the performance of the optimized model outperforms the baseline model.