ASPEN: An Efficient Algorithm for Data Redistribution Between Producer and Consumer Grids

Research output: Chapter in Book/Report/Conference proceedingConference Contribution (Conference Proceeding)

1 Citation (Scopus)
103 Downloads (Pure)

Abstract

HPC applications and libraries have frequently moved parallel data from one distribution scheme to another, for reasons of performance. In modern times, a resurgence of interest in this data redistribution problem has emerged due to the need to relocate data distributed across one Producer grid onto a different distribution scheme across a Consumer grid. In this paper, we study the efficient algorithms to perform redistribution, and show how the best methods from the literature are still dependent on the number of processors in both grids. We describe a new algorithm ASPEN that exploits more cyclic patterns and relations in the distribution, is not dependent on the total number of processors and is thus well suited for use in a workflow management systems. We describe a preliminary implementation of the algorithm within such a workflow system and show performance results that indicate a significant performance benefit in data redistribution generation.
Original languageEnglish
Title of host publicationEuro-Par 2018: Parallel Processing Workshops
Subtitle of host publicationEuro-Par 2018 International Workshops, Turin, Italy, August 27-28, 2018, Revised Selected Papers
PublisherSpringer, Cham
Pages171-182
Number of pages12
ISBN (Electronic)9783030105495
ISBN (Print)9783030105488
DOIs
Publication statusPublished - 31 Dec 2018

Publication series

NameLecture Notes in Computer Science
Volume11339
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Keywords

  • Data Distribution
  • Redistribution
  • Data Placement
  • Data Locality
  • Memory Layout
  • Communication Pattern
  • Parallel Programming
  • Distributed Memory

Cite this