An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds
The Low Frequency Array (LOFAR) radio telescope is an international aperture synthesis radio telescope used to study the Universe at low frequencies. One of the goals of the LOFAR telescope is to conduct deep wide-field surveys. Here we will discuss a framework for the processing of the LOFAR Two Me...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Journal Article |
| Published: |
SPIE - Internatioal Society for Optocal Engineering
2017
|
| Online Access: | http://hdl.handle.net/20.500.11937/74132 |
| _version_ | 1848763188992016384 |
|---|---|
| author | Mechev, A. Oonk, J. Danezi, A. Shimwell, T. Schrijvers, C. Intema, Hubertus Plaat, A. Rttgering, H. |
| author_facet | Mechev, A. Oonk, J. Danezi, A. Shimwell, T. Schrijvers, C. Intema, Hubertus Plaat, A. Rttgering, H. |
| author_sort | Mechev, A. |
| building | Curtin Institutional Repository |
| collection | Online Access |
| description | The Low Frequency Array (LOFAR) radio telescope is an international aperture synthesis radio telescope used to study the Universe at low frequencies. One of the goals of the LOFAR telescope is to conduct deep wide-field surveys. Here we will discuss a framework for the processing of the LOFAR Two Meter Sky Survey (LoTSS). This survey will produce close to 50 PB of data within five years. These data rates require processing at locations with high-speed access to the archived data. To complete the LoTSS project, the processing software needs to be made portable and moved to clusters with a high bandwidth connection to the data archive. This work presents a framework that makes the LOFAR software portable, and is used to scale out LOFAR data reduction. Previous work was successful in pre-processing LOFAR data on a cluster of isolated nodes. This framework builds upon it and and is currently operational. It is designed to be portable, scalable, automated and general. This paper describes its design and high level operation and the initial results processing LoTSS data. |
| first_indexed | 2025-11-14T10:59:30Z |
| format | Journal Article |
| id | curtin-20.500.11937-74132 |
| institution | Curtin University Malaysia |
| institution_category | Local University |
| last_indexed | 2025-11-14T10:59:30Z |
| publishDate | 2017 |
| publisher | SPIE - Internatioal Society for Optocal Engineering |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | curtin-20.500.11937-741322019-06-25T06:09:58Z An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds Mechev, A. Oonk, J. Danezi, A. Shimwell, T. Schrijvers, C. Intema, Hubertus Plaat, A. Rttgering, H. The Low Frequency Array (LOFAR) radio telescope is an international aperture synthesis radio telescope used to study the Universe at low frequencies. One of the goals of the LOFAR telescope is to conduct deep wide-field surveys. Here we will discuss a framework for the processing of the LOFAR Two Meter Sky Survey (LoTSS). This survey will produce close to 50 PB of data within five years. These data rates require processing at locations with high-speed access to the archived data. To complete the LoTSS project, the processing software needs to be made portable and moved to clusters with a high bandwidth connection to the data archive. This work presents a framework that makes the LOFAR software portable, and is used to scale out LOFAR data reduction. Previous work was successful in pre-processing LOFAR data on a cluster of isolated nodes. This framework builds upon it and and is currently operational. It is designed to be portable, scalable, automated and general. This paper describes its design and high level operation and the initial results processing LoTSS data. 2017 Journal Article http://hdl.handle.net/20.500.11937/74132 http://creativecommons.org/licenses/by-nc-nd/4.0/ SPIE - Internatioal Society for Optocal Engineering fulltext |
| spellingShingle | Mechev, A. Oonk, J. Danezi, A. Shimwell, T. Schrijvers, C. Intema, Hubertus Plaat, A. Rttgering, H. An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds |
| title | An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds |
| title_full | An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds |
| title_fullStr | An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds |
| title_full_unstemmed | An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds |
| title_short | An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds |
| title_sort | automated scalable framework for distributing radio astronomy processing across clusters and clouds |
| url | http://hdl.handle.net/20.500.11937/74132 |