ASPEN: An Efficient Algorithm for Data Redistribution Between Producer and Consumer Grids

Clement Foyer; A Tate; Simon McIntosh-Smith

doi:10.1007/978-3-030-10549-5_14

ASPEN: An Efficient Algorithm for Data Redistribution Between Producer and Consumer Grids

Clement Foyer, A Tate, Simon McIntosh-Smith

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

2 Citations (Scopus)

162 Downloads (Pure)

Abstract

HPC applications and libraries have frequently moved parallel data from one distribution scheme to another, for reasons of performance. In modern times, a resurgence of interest in this data redistribution problem has emerged due to the need to relocate data distributed across one Producer grid onto a different distribution scheme across a Consumer grid. In this paper, we study the efficient algorithms to perform redistribution, and show how the best methods from the literature are still dependent on the number of processors in both grids. We describe a new algorithm ASPEN that exploits more cyclic patterns and relations in the distribution, is not dependent on the total number of processors and is thus well suited for use in a workflow management systems. We describe a preliminary implementation of the algorithm within such a workflow system and show performance results that indicate a significant performance benefit in data redistribution generation.

Original language	English
Title of host publication	Euro-Par 2018: Parallel Processing Workshops
Subtitle of host publication	Euro-Par 2018 International Workshops, Turin, Italy, August 27-28, 2018, Revised Selected Papers
Publisher	Springer, Cham
Pages	171-182
Number of pages	12
ISBN (Electronic)	9783030105495
ISBN (Print)	9783030105488
DOIs	https://doi.org/10.1007/978-3-030-10549-5_14
Publication status	Published - 31 Dec 2018

Publication series

Name	Lecture Notes in Computer Science
Volume	11339
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Keywords

Data Distribution
Redistribution
Data Placement
Data Locality
Memory Layout
Communication Pattern
Parallel Programming
Distributed Memory

Access to Document

10.1007/978-3-030-10549-5_14

Full-text PDF (accepted author manuscript)
This is the author accepted manuscript (AAM). The final published version (version of record) is available online via Springer at https://link.springer.com/chapter/10.1007%2F978-3-030-10549-5_14 . Please refer to any applicable terms of use of the publisher.
Accepted author manuscript, 646 KB

Handle.net

Persistent link

Cite this

Foyer, C., Tate, A., & McIntosh-Smith, S. (2018). ASPEN: An Efficient Algorithm for Data Redistribution Between Producer and Consumer Grids. In Euro-Par 2018: Parallel Processing Workshops: Euro-Par 2018 International Workshops, Turin, Italy, August 27-28, 2018, Revised Selected Papers (pp. 171-182). (Lecture Notes in Computer Science; Vol. 11339). Springer, Cham. https://doi.org/10.1007/978-3-030-10549-5_14

@inproceedings{38b50a19d36142e9928e20f22aae1484,

title = "ASPEN: An Efficient Algorithm for Data Redistribution Between Producer and Consumer Grids",

abstract = "HPC applications and libraries have frequently moved parallel data from one distribution scheme to another, for reasons of performance. In modern times, a resurgence of interest in this data redistribution problem has emerged due to the need to relocate data distributed across one Producer grid onto a different distribution scheme across a Consumer grid. In this paper, we study the efficient algorithms to perform redistribution, and show how the best methods from the literature are still dependent on the number of processors in both grids. We describe a new algorithm ASPEN that exploits more cyclic patterns and relations in the distribution, is not dependent on the total number of processors and is thus well suited for use in a workflow management systems. We describe a preliminary implementation of the algorithm within such a workflow system and show performance results that indicate a significant performance benefit in data redistribution generation.",

keywords = "Data Distribution, Redistribution, Data Placement, Data Locality, Memory Layout, Communication Pattern, Parallel Programming, Distributed Memory",

author = "Clement Foyer and A Tate and Simon McIntosh-Smith",

year = "2018",

month = dec,

day = "31",

doi = "10.1007/978-3-030-10549-5_14",

language = "English",

isbn = "9783030105488",

series = "Lecture Notes in Computer Science",

publisher = "Springer, Cham",

pages = "171--182",

booktitle = "Euro-Par 2018: Parallel Processing Workshops",

address = "Switzerland",

}

Foyer, C, Tate, A & McIntosh-Smith, S 2018, ASPEN: An Efficient Algorithm for Data Redistribution Between Producer and Consumer Grids. in Euro-Par 2018: Parallel Processing Workshops: Euro-Par 2018 International Workshops, Turin, Italy, August 27-28, 2018, Revised Selected Papers. Lecture Notes in Computer Science, vol. 11339, Springer, Cham, pp. 171-182. https://doi.org/10.1007/978-3-030-10549-5_14

ASPEN: An Efficient Algorithm for Data Redistribution Between Producer and Consumer Grids. / Foyer, Clement; Tate, A; McIntosh-Smith, Simon.
Euro-Par 2018: Parallel Processing Workshops: Euro-Par 2018 International Workshops, Turin, Italy, August 27-28, 2018, Revised Selected Papers. Springer, Cham, 2018. p. 171-182 (Lecture Notes in Computer Science; Vol. 11339).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - ASPEN

T2 - An Efficient Algorithm for Data Redistribution Between Producer and Consumer Grids

AU - Foyer, Clement

AU - Tate, A

AU - McIntosh-Smith, Simon

PY - 2018/12/31

Y1 - 2018/12/31

N2 - HPC applications and libraries have frequently moved parallel data from one distribution scheme to another, for reasons of performance. In modern times, a resurgence of interest in this data redistribution problem has emerged due to the need to relocate data distributed across one Producer grid onto a different distribution scheme across a Consumer grid. In this paper, we study the efficient algorithms to perform redistribution, and show how the best methods from the literature are still dependent on the number of processors in both grids. We describe a new algorithm ASPEN that exploits more cyclic patterns and relations in the distribution, is not dependent on the total number of processors and is thus well suited for use in a workflow management systems. We describe a preliminary implementation of the algorithm within such a workflow system and show performance results that indicate a significant performance benefit in data redistribution generation.

AB - HPC applications and libraries have frequently moved parallel data from one distribution scheme to another, for reasons of performance. In modern times, a resurgence of interest in this data redistribution problem has emerged due to the need to relocate data distributed across one Producer grid onto a different distribution scheme across a Consumer grid. In this paper, we study the efficient algorithms to perform redistribution, and show how the best methods from the literature are still dependent on the number of processors in both grids. We describe a new algorithm ASPEN that exploits more cyclic patterns and relations in the distribution, is not dependent on the total number of processors and is thus well suited for use in a workflow management systems. We describe a preliminary implementation of the algorithm within such a workflow system and show performance results that indicate a significant performance benefit in data redistribution generation.

KW - Data Distribution

KW - Redistribution

KW - Data Placement

KW - Data Locality

KW - Memory Layout

KW - Communication Pattern

KW - Parallel Programming

KW - Distributed Memory

U2 - 10.1007/978-3-030-10549-5_14

DO - 10.1007/978-3-030-10549-5_14

M3 - Conference Contribution (Conference Proceeding)

SN - 9783030105488

T3 - Lecture Notes in Computer Science

SP - 171

EP - 182

BT - Euro-Par 2018: Parallel Processing Workshops

PB - Springer, Cham

ER -

Foyer C, Tate A, McIntosh-Smith S. ASPEN: An Efficient Algorithm for Data Redistribution Between Producer and Consumer Grids. In Euro-Par 2018: Parallel Processing Workshops: Euro-Par 2018 International Workshops, Turin, Italy, August 27-28, 2018, Revised Selected Papers. Springer, Cham. 2018. p. 171-182. (Lecture Notes in Computer Science). doi: 10.1007/978-3-030-10549-5_14

ASPEN: An Efficient Algorithm for Data Redistribution Between Producer and Consumer Grids

Abstract

Publication series

Keywords

Access to Document

Handle.net

Fingerprint

Cite this