ASPEN: An Efficient Algorithm for Data Redistribution Between Producer and Consumer Grids

Clement Foyer, A Tate, Simon McIntosh-Smith

Research output: Chapter in Book/Report/Conference proceedingConference Contribution (Conference Proceeding)

2 Citations (Scopus)
134 Downloads (Pure)


HPC applications and libraries have frequently moved parallel data from one distribution scheme to another, for reasons of performance. In modern times, a resurgence of interest in this data redistribution problem has emerged due to the need to relocate data distributed across one Producer grid onto a different distribution scheme across a Consumer grid. In this paper, we study the efficient algorithms to perform redistribution, and show how the best methods from the literature are still dependent on the number of processors in both grids. We describe a new algorithm ASPEN that exploits more cyclic patterns and relations in the distribution, is not dependent on the total number of processors and is thus well suited for use in a workflow management systems. We describe a preliminary implementation of the algorithm within such a workflow system and show performance results that indicate a significant performance benefit in data redistribution generation.
Original languageEnglish
Title of host publicationEuro-Par 2018: Parallel Processing Workshops
Subtitle of host publicationEuro-Par 2018 International Workshops, Turin, Italy, August 27-28, 2018, Revised Selected Papers
PublisherSpringer, Cham
Number of pages12
ISBN (Electronic)9783030105495
ISBN (Print)9783030105488
Publication statusPublished - 31 Dec 2018

Publication series

NameLecture Notes in Computer Science
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


  • Data Distribution
  • Redistribution
  • Data Placement
  • Data Locality
  • Memory Layout
  • Communication Pattern
  • Parallel Programming
  • Distributed Memory


Dive into the research topics of 'ASPEN: An Efficient Algorithm for Data Redistribution Between Producer and Consumer Grids'. Together they form a unique fingerprint.

Cite this