SEQ2SEQ++: a multitasking-based seq2seq model to generate meaningful and relevant answers
Question-answering chatbots have tremendous potential to complement humans in various fields. They are implemented using either rule-based or machine learning-based systems. Unlike the former, machine learning-based chatbots are more scalable. Sequence-to-sequence (Seq2Seq) learning is one of the mo...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Institute of Electrical and Electronics Engineers
2021
|
| Online Access: | http://psasir.upm.edu.my/id/eprint/95045/ http://psasir.upm.edu.my/id/eprint/95045/1/SEQ2SEQ%2B%2B%20a%20multitasking-based%20seq2seq%20model%20to%20generate%20meaningful%20and%20relevant%20answers.pdf |
| _version_ | 1848862059867930624 |
|---|---|
| author | Palasundram, Kulothunkan Mohd Sharef, Nurfadhlina Kasmiran, Khairul Azhar Azman, Azreen |
| author_facet | Palasundram, Kulothunkan Mohd Sharef, Nurfadhlina Kasmiran, Khairul Azhar Azman, Azreen |
| author_sort | Palasundram, Kulothunkan |
| building | UPM Institutional Repository |
| collection | Online Access |
| description | Question-answering chatbots have tremendous potential to complement humans in various fields. They are implemented using either rule-based or machine learning-based systems. Unlike the former, machine learning-based chatbots are more scalable. Sequence-to-sequence (Seq2Seq) learning is one of the most popular approaches in machine learning-based chatbots and has shown remarkable progress since its introduction in 2014. However, chatbots based on Seq2Seq learning show a weakness in that it tends to generate answers that can be generic and inconsistent with the questions, thereby becoming meaningless and, therefore, may lower the chatbot adoption rate. This weakness can be attributed to three issues: question encoder overfit, answer generation overfit, and language model influence. Several recent methods utilize multitask learning (MTL) to address this weakness. However, the existing MTL models show very little improvement over single-task learning, wherein they still generate generic and inconsistent answers. This paper presents a novel approach to MTL for the Seq2Seq learning model called SEQ2SEQ++, which comprises a multifunctional encoder, an answer decoder, an answer encoder, and a ternary classifier. Additionally, SEQ2SEQ++ utilizes a dynamic tasks loss weight mechanism for MTL loss calculation and a novel attention mechanism called the comprehensive attention mechanism. Experiments on NarrativeQA and SQuAD datasets were conducted to gauge the performance of the proposed model in comparison with two recently proposed models. The experimental results show that SEQ2SEQ++ yields noteworthy improvements over the two models on bilingual evaluation understudy, word error rate, and Distinct-2 metrics. |
| first_indexed | 2025-11-15T13:11:01Z |
| format | Article |
| id | upm-95045 |
| institution | Universiti Putra Malaysia |
| institution_category | Local University |
| language | English |
| last_indexed | 2025-11-15T13:11:01Z |
| publishDate | 2021 |
| publisher | Institute of Electrical and Electronics Engineers |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | upm-950452023-01-05T08:53:16Z http://psasir.upm.edu.my/id/eprint/95045/ SEQ2SEQ++: a multitasking-based seq2seq model to generate meaningful and relevant answers Palasundram, Kulothunkan Mohd Sharef, Nurfadhlina Kasmiran, Khairul Azhar Azman, Azreen Question-answering chatbots have tremendous potential to complement humans in various fields. They are implemented using either rule-based or machine learning-based systems. Unlike the former, machine learning-based chatbots are more scalable. Sequence-to-sequence (Seq2Seq) learning is one of the most popular approaches in machine learning-based chatbots and has shown remarkable progress since its introduction in 2014. However, chatbots based on Seq2Seq learning show a weakness in that it tends to generate answers that can be generic and inconsistent with the questions, thereby becoming meaningless and, therefore, may lower the chatbot adoption rate. This weakness can be attributed to three issues: question encoder overfit, answer generation overfit, and language model influence. Several recent methods utilize multitask learning (MTL) to address this weakness. However, the existing MTL models show very little improvement over single-task learning, wherein they still generate generic and inconsistent answers. This paper presents a novel approach to MTL for the Seq2Seq learning model called SEQ2SEQ++, which comprises a multifunctional encoder, an answer decoder, an answer encoder, and a ternary classifier. Additionally, SEQ2SEQ++ utilizes a dynamic tasks loss weight mechanism for MTL loss calculation and a novel attention mechanism called the comprehensive attention mechanism. Experiments on NarrativeQA and SQuAD datasets were conducted to gauge the performance of the proposed model in comparison with two recently proposed models. The experimental results show that SEQ2SEQ++ yields noteworthy improvements over the two models on bilingual evaluation understudy, word error rate, and Distinct-2 metrics. Institute of Electrical and Electronics Engineers 2021-12-06 Article PeerReviewed text en http://psasir.upm.edu.my/id/eprint/95045/1/SEQ2SEQ%2B%2B%20a%20multitasking-based%20seq2seq%20model%20to%20generate%20meaningful%20and%20relevant%20answers.pdf Palasundram, Kulothunkan and Mohd Sharef, Nurfadhlina and Kasmiran, Khairul Azhar and Azman, Azreen (2021) SEQ2SEQ++: a multitasking-based seq2seq model to generate meaningful and relevant answers. IEEE Access, 9. 164949 - 164975. ISSN 2169-3536 https://ieeexplore.ieee.org/document/9638628 10.1109/ACCESS.2021.3133495 |
| spellingShingle | Palasundram, Kulothunkan Mohd Sharef, Nurfadhlina Kasmiran, Khairul Azhar Azman, Azreen SEQ2SEQ++: a multitasking-based seq2seq model to generate meaningful and relevant answers |
| title | SEQ2SEQ++: a multitasking-based seq2seq model to generate meaningful and relevant answers |
| title_full | SEQ2SEQ++: a multitasking-based seq2seq model to generate meaningful and relevant answers |
| title_fullStr | SEQ2SEQ++: a multitasking-based seq2seq model to generate meaningful and relevant answers |
| title_full_unstemmed | SEQ2SEQ++: a multitasking-based seq2seq model to generate meaningful and relevant answers |
| title_short | SEQ2SEQ++: a multitasking-based seq2seq model to generate meaningful and relevant answers |
| title_sort | seq2seq++: a multitasking-based seq2seq model to generate meaningful and relevant answers |
| url | http://psasir.upm.edu.my/id/eprint/95045/ http://psasir.upm.edu.my/id/eprint/95045/ http://psasir.upm.edu.my/id/eprint/95045/ http://psasir.upm.edu.my/id/eprint/95045/1/SEQ2SEQ%2B%2B%20a%20multitasking-based%20seq2seq%20model%20to%20generate%20meaningful%20and%20relevant%20answers.pdf |