Scalability model for the LOFAR direction independent pipeline
LOFAR is a leading aperture synthesis telescope operated in the Netherlands with stations across Europe. The LOFAR Two-meter Sky Survey (LoTSS) will produce more than 3000 14 TB data sets, mapping the entire northern sky at low frequencies. The data produced by this survey is important for understan...
| Main Authors: | , , , , , |
|---|---|
| Format: | Journal Article |
| Published: |
2019
|
| Online Access: | http://hdl.handle.net/20.500.11937/76016 |
| _version_ | 1848763604573093888 |
|---|---|
| author | Mechev, A.P. Shimwell, T.W. Plaat, A. Intema, Huib Varbanescu, A.L. Rottgering, H.J.A. |
| author_facet | Mechev, A.P. Shimwell, T.W. Plaat, A. Intema, Huib Varbanescu, A.L. Rottgering, H.J.A. |
| author_sort | Mechev, A.P. |
| building | Curtin Institutional Repository |
| collection | Online Access |
| description | LOFAR is a leading aperture synthesis telescope operated in the Netherlands with stations across Europe. The LOFAR Two-meter Sky Survey (LoTSS) will produce more than 3000 14 TB data sets, mapping the entire northern sky at low frequencies. The data produced by this survey is important for understanding the formation and evolution of galaxies, supermassive black holes and other astronomical phenomena. All of the LoTSS data needs to be processed by the LOFAR Direction Independent (DI) pipeline, prefactor. Understanding the performance of this pipeline is important when trying to optimize the throughput for large projects, such as LoTSS and other deep surveys. Making a model of its completion time will enable us to predict the time taken to process large data sets, optimize our parameter choices, help schedule other LOFAR processing services, and predict processing time for future large radio telescopes. We tested the prefactor pipeline by scaling several parameters, notably number of CPUs, data size and size of calibration sky model. We present these results as a comprehensive model which will be used to predict processing time for a wide range of processing parameters. We also discover that smaller calibration models lead to significantly faster calibration times, while the calibration results do not significantly degrade in quality. Finally, we validate the model and compare predictions with production runs from the past six months, quantifying the performance penalties incurred by processing on a shared cluster. We conclude by noting the utility of the results and model for the LoTSS Survey, LOFAR as a whole and for other telescopes. |
| first_indexed | 2025-11-14T11:06:06Z |
| format | Journal Article |
| id | curtin-20.500.11937-76016 |
| institution | Curtin University Malaysia |
| institution_category | Local University |
| last_indexed | 2025-11-14T11:06:06Z |
| publishDate | 2019 |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | curtin-20.500.11937-760162019-07-23T03:00:53Z Scalability model for the LOFAR direction independent pipeline Mechev, A.P. Shimwell, T.W. Plaat, A. Intema, Huib Varbanescu, A.L. Rottgering, H.J.A. LOFAR is a leading aperture synthesis telescope operated in the Netherlands with stations across Europe. The LOFAR Two-meter Sky Survey (LoTSS) will produce more than 3000 14 TB data sets, mapping the entire northern sky at low frequencies. The data produced by this survey is important for understanding the formation and evolution of galaxies, supermassive black holes and other astronomical phenomena. All of the LoTSS data needs to be processed by the LOFAR Direction Independent (DI) pipeline, prefactor. Understanding the performance of this pipeline is important when trying to optimize the throughput for large projects, such as LoTSS and other deep surveys. Making a model of its completion time will enable us to predict the time taken to process large data sets, optimize our parameter choices, help schedule other LOFAR processing services, and predict processing time for future large radio telescopes. We tested the prefactor pipeline by scaling several parameters, notably number of CPUs, data size and size of calibration sky model. We present these results as a comprehensive model which will be used to predict processing time for a wide range of processing parameters. We also discover that smaller calibration models lead to significantly faster calibration times, while the calibration results do not significantly degrade in quality. Finally, we validate the model and compare predictions with production runs from the past six months, quantifying the performance penalties incurred by processing on a shared cluster. We conclude by noting the utility of the results and model for the LoTSS Survey, LOFAR as a whole and for other telescopes. 2019 Journal Article http://hdl.handle.net/20.500.11937/76016 10.1016/j.ascom.2019.100293 restricted |
| spellingShingle | Mechev, A.P. Shimwell, T.W. Plaat, A. Intema, Huib Varbanescu, A.L. Rottgering, H.J.A. Scalability model for the LOFAR direction independent pipeline |
| title | Scalability model for the LOFAR direction independent pipeline |
| title_full | Scalability model for the LOFAR direction independent pipeline |
| title_fullStr | Scalability model for the LOFAR direction independent pipeline |
| title_full_unstemmed | Scalability model for the LOFAR direction independent pipeline |
| title_short | Scalability model for the LOFAR direction independent pipeline |
| title_sort | scalability model for the lofar direction independent pipeline |
| url | http://hdl.handle.net/20.500.11937/76016 |