Distributed indexing: performance analysis of Solr, Terrier and Katta Information Retrievals
Information Retrieval (IR) systems are currently facing a continuous challenge due to the increasing size of datasets. Extremely, large data from different aspects is gathered each day resulting in huge increase in the scale of the raw data available across the internet. Indexing, as the main functi...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
culty of Computer Science and Information Technology, University of Malaya
2018
|
| Online Access: | http://psasir.upm.edu.my/id/eprint/72313/ http://psasir.upm.edu.my/id/eprint/72313/1/Distributed%20indexing.pdf |
| _version_ | 1848857088180092928 |
|---|---|
| author | Aldailamy, Ali Y. Abdul Hamid, Nor Asila Wati Abdulkarem, Mohammed |
| author_facet | Aldailamy, Ali Y. Abdul Hamid, Nor Asila Wati Abdulkarem, Mohammed |
| author_sort | Aldailamy, Ali Y. |
| building | UPM Institutional Repository |
| collection | Online Access |
| description | Information Retrieval (IR) systems are currently facing a continuous challenge due to the increasing size of datasets. Extremely, large data from different aspects is gathered each day resulting in huge increase in the scale of the raw data available across the internet. Indexing, as the main function of IR systems, is becoming a time-consuming problem. Therefore, efficient indexing of a large volume of data is now a critical requirement of modern IR systems. High performance indexing is performed nowadays over the use of MapReduce programming model. MapReduce is a programming paradigm that enables massive processing and large collections distribution across multiple (hundreds or thousands) commodity computers. To shed some light on this issue, this paper presents a detailed performance analysis of distributed indexing for Solr, Katta and Terrier with the context of MapReduce. In particular, this study compares and analyzes the distributed indexing performance of three frameworks using 1GB, 3GB, 6GB, and 9GB subsets of TREC dataset as the processing power increase. The experiments measure the indexing average time, then throughput, speedup, and efficiency of indexing process. The results show that, Terrier performance is the best in the presence of large collections and scalable processing power. While, Solr performance is the best when having limited computing power and small document collections. Finally, the experimental results show that, Katta produced the worst indexing average time among the three frameworks but its speedup scales linearly with processing power and collection size. |
| first_indexed | 2025-11-15T11:51:59Z |
| format | Article |
| id | upm-72313 |
| institution | Universiti Putra Malaysia |
| institution_category | Local University |
| language | English |
| last_indexed | 2025-11-15T11:51:59Z |
| publishDate | 2018 |
| publisher | culty of Computer Science and Information Technology, University of Malaya |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | upm-723132020-05-04T01:34:52Z http://psasir.upm.edu.my/id/eprint/72313/ Distributed indexing: performance analysis of Solr, Terrier and Katta Information Retrievals Aldailamy, Ali Y. Abdul Hamid, Nor Asila Wati Abdulkarem, Mohammed Information Retrieval (IR) systems are currently facing a continuous challenge due to the increasing size of datasets. Extremely, large data from different aspects is gathered each day resulting in huge increase in the scale of the raw data available across the internet. Indexing, as the main function of IR systems, is becoming a time-consuming problem. Therefore, efficient indexing of a large volume of data is now a critical requirement of modern IR systems. High performance indexing is performed nowadays over the use of MapReduce programming model. MapReduce is a programming paradigm that enables massive processing and large collections distribution across multiple (hundreds or thousands) commodity computers. To shed some light on this issue, this paper presents a detailed performance analysis of distributed indexing for Solr, Katta and Terrier with the context of MapReduce. In particular, this study compares and analyzes the distributed indexing performance of three frameworks using 1GB, 3GB, 6GB, and 9GB subsets of TREC dataset as the processing power increase. The experiments measure the indexing average time, then throughput, speedup, and efficiency of indexing process. The results show that, Terrier performance is the best in the presence of large collections and scalable processing power. While, Solr performance is the best when having limited computing power and small document collections. Finally, the experimental results show that, Katta produced the worst indexing average time among the three frameworks but its speedup scales linearly with processing power and collection size. culty of Computer Science and Information Technology, University of Malaya 2018-12 Article NonPeerReviewed text en http://psasir.upm.edu.my/id/eprint/72313/1/Distributed%20indexing.pdf Aldailamy, Ali Y. and Abdul Hamid, Nor Asila Wati and Abdulkarem, Mohammed (2018) Distributed indexing: performance analysis of Solr, Terrier and Katta Information Retrievals. Malaysian Journal of Computer Science (spec. 2018). 87- 104. ISSN 0127-9084 https://ejournal.um.edu.my/index.php/MJCS/article/view/15490 10.22452/mjcs.sp2018no1.7 |
| spellingShingle | Aldailamy, Ali Y. Abdul Hamid, Nor Asila Wati Abdulkarem, Mohammed Distributed indexing: performance analysis of Solr, Terrier and Katta Information Retrievals |
| title | Distributed indexing: performance analysis of Solr, Terrier and Katta Information Retrievals |
| title_full | Distributed indexing: performance analysis of Solr, Terrier and Katta Information Retrievals |
| title_fullStr | Distributed indexing: performance analysis of Solr, Terrier and Katta Information Retrievals |
| title_full_unstemmed | Distributed indexing: performance analysis of Solr, Terrier and Katta Information Retrievals |
| title_short | Distributed indexing: performance analysis of Solr, Terrier and Katta Information Retrievals |
| title_sort | distributed indexing: performance analysis of solr, terrier and katta information retrievals |
| url | http://psasir.upm.edu.my/id/eprint/72313/ http://psasir.upm.edu.my/id/eprint/72313/ http://psasir.upm.edu.my/id/eprint/72313/ http://psasir.upm.edu.my/id/eprint/72313/1/Distributed%20indexing.pdf |