Enhancements to the sequence-to-sequence-based natural answer generation models
There is a great interest shown by academic researchers to continuously improve the sequence-to-sequence (Seq2Seq) model for natural answer generation (NAG) in chatbots. The Seq2Seq model shows a weakness whereby the model tends to generate answers that are generic, meaningless and inconsistent with...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Institute of Electrical and Electronics Engineers
2020
|
| Online Access: | http://psasir.upm.edu.my/id/eprint/88819/ http://psasir.upm.edu.my/id/eprint/88819/1/BOT.pdf |
| _version_ | 1848860702300700672 |
|---|---|
| author | Palasundram, Kulothunkan Mohd Sharef, Nurfadhlina Kasmiran, Khairul Azhar Azman, Azreen |
| author_facet | Palasundram, Kulothunkan Mohd Sharef, Nurfadhlina Kasmiran, Khairul Azhar Azman, Azreen |
| author_sort | Palasundram, Kulothunkan |
| building | UPM Institutional Repository |
| collection | Online Access |
| description | There is a great interest shown by academic researchers to continuously improve the sequence-to-sequence (Seq2Seq) model for natural answer generation (NAG) in chatbots. The Seq2Seq model shows a weakness whereby the model tends to generate answers that are generic, meaningless and inconsistent with the questions. However, a comprehensive literature review on the factors contributing to the weakness and potential solutions are still missing. Therefore, this review article fills the gap by reviewing Seq2Seq based natural answer generation-based literature to identify those factors and proposed methods to address the weakness. This literature review identified several factors such as input question is not sufficient to determine a meaningful output, usage of cross-entropy function as the loss function during training, infrequent words in training data, language model influence which generates answers not relevant to the question, utilization of teacher forcing method during training which results in exposure bias, long sentences and inability to consider dialogue history as the factors. Additionally, this literature review also identified and reviewed the methods proposed to address the weakness such as utilizing additional embedding and encoders, using different loss functions and training approaches, as well as utilizing other mechanisms like copying source word(s) and paying attention to a certain portion of the input. For discussion, these methods are categorized into four broad categories which are Structural Modifications, Augmented Learning, Beam Search and Complementary Mechanisms. Additionally, the paper highlights unexplored areas in Seq2Seq modeling and proposes potential future works for natural answer generation. |
| first_indexed | 2025-11-15T12:49:26Z |
| format | Article |
| id | upm-88819 |
| institution | Universiti Putra Malaysia |
| institution_category | Local University |
| language | English |
| last_indexed | 2025-11-15T12:49:26Z |
| publishDate | 2020 |
| publisher | Institute of Electrical and Electronics Engineers |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | upm-888192021-10-05T23:18:38Z http://psasir.upm.edu.my/id/eprint/88819/ Enhancements to the sequence-to-sequence-based natural answer generation models Palasundram, Kulothunkan Mohd Sharef, Nurfadhlina Kasmiran, Khairul Azhar Azman, Azreen There is a great interest shown by academic researchers to continuously improve the sequence-to-sequence (Seq2Seq) model for natural answer generation (NAG) in chatbots. The Seq2Seq model shows a weakness whereby the model tends to generate answers that are generic, meaningless and inconsistent with the questions. However, a comprehensive literature review on the factors contributing to the weakness and potential solutions are still missing. Therefore, this review article fills the gap by reviewing Seq2Seq based natural answer generation-based literature to identify those factors and proposed methods to address the weakness. This literature review identified several factors such as input question is not sufficient to determine a meaningful output, usage of cross-entropy function as the loss function during training, infrequent words in training data, language model influence which generates answers not relevant to the question, utilization of teacher forcing method during training which results in exposure bias, long sentences and inability to consider dialogue history as the factors. Additionally, this literature review also identified and reviewed the methods proposed to address the weakness such as utilizing additional embedding and encoders, using different loss functions and training approaches, as well as utilizing other mechanisms like copying source word(s) and paying attention to a certain portion of the input. For discussion, these methods are categorized into four broad categories which are Structural Modifications, Augmented Learning, Beam Search and Complementary Mechanisms. Additionally, the paper highlights unexplored areas in Seq2Seq modeling and proposes potential future works for natural answer generation. Institute of Electrical and Electronics Engineers 2020 Article PeerReviewed text en http://psasir.upm.edu.my/id/eprint/88819/1/BOT.pdf Palasundram, Kulothunkan and Mohd Sharef, Nurfadhlina and Kasmiran, Khairul Azhar and Azman, Azreen (2020) Enhancements to the sequence-to-sequence-based natural answer generation models. IEEE Access, 8. 45738 - 45752. ISSN 2169-3536 https://ieeexplore.ieee.org/document/9025265 10.1109/ACCESS.2020.2978551 |
| spellingShingle | Palasundram, Kulothunkan Mohd Sharef, Nurfadhlina Kasmiran, Khairul Azhar Azman, Azreen Enhancements to the sequence-to-sequence-based natural answer generation models |
| title | Enhancements to the sequence-to-sequence-based natural answer generation models |
| title_full | Enhancements to the sequence-to-sequence-based natural answer generation models |
| title_fullStr | Enhancements to the sequence-to-sequence-based natural answer generation models |
| title_full_unstemmed | Enhancements to the sequence-to-sequence-based natural answer generation models |
| title_short | Enhancements to the sequence-to-sequence-based natural answer generation models |
| title_sort | enhancements to the sequence-to-sequence-based natural answer generation models |
| url | http://psasir.upm.edu.my/id/eprint/88819/ http://psasir.upm.edu.my/id/eprint/88819/ http://psasir.upm.edu.my/id/eprint/88819/ http://psasir.upm.edu.my/id/eprint/88819/1/BOT.pdf |