Enabling System Wide Shared Memory for Performance Improvement in PyCOMPSs Applications

Clement M Foyer, Javier Conejero, Jorge Ejarque, Rosa M. Badia, Adrian Tate, Simon N Mcintosh-Smith

Research output: Chapter in Book/Report/Conference proceedingConference Contribution (Conference Proceeding)

26 Downloads (Pure)


Python has been gaining some traction for years in the world of scientific applications. However, the high-level abstraction it provides may not allow the developer to use the machines to their peak performance. To address this, multiple strategies, sometimes complementary, have been developed to enrich the software ecosystem either by relying on additional libraries dedicated to efficient computation (e.g., NumPy) or by providing a framework to better use HPC scale infrastructures (e.g., PyCOMPSs).In this paper, we present a Python extension based on SharedArray that enables the support of system-provided shared memory and its integration into the PyCOMPSs programming model as an example of integration to a complex Python environment. We also evaluate the impact such a tool may have on performance in two types of distributed execution-flows, one for linear algebra with a blocked matrix multiplication application and the other in the context of data-clustering with a k-means application. We show that with very little modification of the original decorator (3 lines of code to be modified) of the task-based application the gain in performance can rise above 40% for tasks relying heavily on data reuse on a distributed environment, especially when loading the data is prominent in the execution time.
Original languageEnglish
Title of host publication2020 IEEE/ACM 9th Workshop on Python for High-Performance and Scientific Computing (PyHPC)
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
ISBN (Electronic)978-0-7381-1086-8
ISBN (Print)978-1-6654-2286-4
Publication statusPublished - 30 Dec 2020


  • memory
  • shared memory
  • task
  • python
  • Parallel programming
  • distributed memory
  • NumPy
  • Data Management
  • Task analysis
  • Runtime
  • Libraries
  • Tools
  • Metadata


Dive into the research topics of 'Enabling System Wide Shared Memory for Performance Improvement in PyCOMPSs Applications'. Together they form a unique fingerprint.

Cite this