CircuitFlow: A Domain Specific Language for Dataflow Programming

Samantha Frohlich; Meng Wang; Riley Evans

doi:10.1007/978-3-030-94479-7_6

CircuitFlow: A Domain Specific Language for Dataflow Programming

Samantha Frohlich, Meng Wang^*, Riley Evans

^*Corresponding author for this work

Bristol Doctoral College

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

6 Citations (Scopus)

108 Downloads (Pure)

Abstract

Dataflow applications, such as machine learning algorithms,
can run for days, making it desirable to have assurances that they will
work correctly. Current tools are not good enough: too often the interactions between tasks are not type-safe, leading to undesirable runtime
errors. This paper presents a new declarative Haskell Embedded DSL
(eDSL) for dataflow programming: CircuitFlow. Defined as a Symmetric Monoidal Preorder (SMP) on data that models dependencies in the
workflow, it has a strong mathematical basis, refocusing on how data
flows through an application, resulting in a more expressive solution
that not only catches errors statically, but also achieves competitive runtime performance. In our preliminary evaluation, CircuitFlow outperforms
the industry-leading Luigi library of Spotify by scaling better with the
number of inputs. The innovative creation of CircuitFlow is also of note,
exemplifying how to create a modular eDSL whose semantics necessitates effects, and where storing complex type information for program
correctness is paramount.

Original language	English
Title of host publication	Practical Aspects of Declarative Languages
Subtitle of host publication	24th International Symposium, PADL 2022, Philadelphia, PA, USA, January 17–18, 2022, Proceedings
Editors	James Cheney, Simona Perri
Publisher	Springer Nature
Pages	79-98
Number of pages	20
ISBN (Electronic)	978-3-030-94479-7
ISBN (Print)	978-3-030-94478-0
DOIs	https://doi.org/10.1007/978-3-030-94479-7_6
Publication status	Published - 31 Jan 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13165 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Bibliographical note

Funding Information:
The work is partly supported by EPSRC Grant EXHIBIT: Expressive High-Level Languages for Bidirectional Transformations (EP/T008911/1) and Royal Society Grant Bidirectional Compiler for Software Evolution (IESR3170104).

Publisher Copyright:
© 2022, Springer Nature Switzerland AG.

Research Groups and Themes

Programming Languages

Keywords

eDSL
Domain-specific languages
Haskell
Dataflow programming

Access to Document

10.1007/978-3-030-94479-7_6

Full-text PDF (author’s accepted manuscript)
This is the author accepted manuscript (AAM). The final published version (version of record) is available online via Springer at https://doi.org/10.1007/978-3-030-94479-7_6 . Please refer to any applicable terms of use of the publisher.
Accepted author manuscript, 290 KB

Handle.net

Persistent link

8030 EPSRC EP/T008911/1 EXHIBIT : Expressive High-Level Languages for Bidirectional Transformations
Wang, M. (Principal Investigator)
1/07/20 → 30/06/23
Project: Research

Cite this

Frohlich, S., Wang, M., & Evans, R. (2022). CircuitFlow: A Domain Specific Language for Dataflow Programming. In J. Cheney, & S. Perri (Eds.), Practical Aspects of Declarative Languages: 24th International Symposium, PADL 2022, Philadelphia, PA, USA, January 17–18, 2022, Proceedings (pp. 79-98). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13165 LNCS). Springer Nature. https://doi.org/10.1007/978-3-030-94479-7_6

Frohlich, Samantha ; Wang, Meng ; Evans, Riley. / CircuitFlow : A Domain Specific Language for Dataflow Programming. Practical Aspects of Declarative Languages: 24th International Symposium, PADL 2022, Philadelphia, PA, USA, January 17–18, 2022, Proceedings. editor / James Cheney ; Simona Perri. Springer Nature, 2022. pp. 79-98 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{527dd05ec094462881cc38481c570ac7,

title = "CircuitFlow: A Domain Specific Language for Dataflow Programming",

abstract = "Dataflow applications, such as machine learning algorithms,can run for days, making it desirable to have assurances that they willwork correctly. Current tools are not good enough: too often the interactions between tasks are not type-safe, leading to undesirable runtimeerrors. This paper presents a new declarative Haskell Embedded DSL(eDSL) for dataflow programming: CircuitFlow. Defined as a Symmetric Monoidal Preorder (SMP) on data that models dependencies in theworkflow, it has a strong mathematical basis, refocusing on how dataflows through an application, resulting in a more expressive solutionthat not only catches errors statically, but also achieves competitive runtime performance. In our preliminary evaluation, CircuitFlow outperformsthe industry-leading Luigi library of Spotify by scaling better with thenumber of inputs. The innovative creation of CircuitFlow is also of note,exemplifying how to create a modular eDSL whose semantics necessitates effects, and where storing complex type information for programcorrectness is paramount.",

keywords = "eDSL, Domain-specific languages, Haskell, Dataflow programming",

author = "Samantha Frohlich and Meng Wang and Riley Evans",

note = "Funding Information: The work is partly supported by EPSRC Grant EXHIBIT: Expressive High-Level Languages for Bidirectional Transformations (EP/T008911/1) and Royal Society Grant Bidirectional Compiler for Software Evolution (IESR3170104). Publisher Copyright: {\textcopyright} 2022, Springer Nature Switzerland AG.",

year = "2022",

month = jan,

day = "31",

doi = "10.1007/978-3-030-94479-7\_6",

language = "English",

isbn = "978-3-030-94478-0",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Nature",

pages = "79--98",

editor = "Cheney, \{James \} and Simona Perri",

booktitle = "Practical Aspects of Declarative Languages",

address = "United Kingdom",

}

Frohlich, S , Wang, M & Evans, R 2022, CircuitFlow: A Domain Specific Language for Dataflow Programming. in J Cheney & S Perri (eds), Practical Aspects of Declarative Languages: 24th International Symposium, PADL 2022, Philadelphia, PA, USA, January 17–18, 2022, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13165 LNCS, Springer Nature, pp. 79-98. https://doi.org/10.1007/978-3-030-94479-7_6

CircuitFlow: A Domain Specific Language for Dataflow Programming. / Frohlich, Samantha ; Wang, Meng; Evans, Riley.
Practical Aspects of Declarative Languages: 24th International Symposium, PADL 2022, Philadelphia, PA, USA, January 17–18, 2022, Proceedings. ed. / James Cheney; Simona Perri. Springer Nature, 2022. p. 79-98 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13165 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - CircuitFlow

T2 - A Domain Specific Language for Dataflow Programming

AU - Frohlich, Samantha

AU - Wang, Meng

AU - Evans, Riley

N1 - Funding Information: The work is partly supported by EPSRC Grant EXHIBIT: Expressive High-Level Languages for Bidirectional Transformations (EP/T008911/1) and Royal Society Grant Bidirectional Compiler for Software Evolution (IESR3170104). Publisher Copyright: © 2022, Springer Nature Switzerland AG.

PY - 2022/1/31

Y1 - 2022/1/31

N2 - Dataflow applications, such as machine learning algorithms,can run for days, making it desirable to have assurances that they willwork correctly. Current tools are not good enough: too often the interactions between tasks are not type-safe, leading to undesirable runtimeerrors. This paper presents a new declarative Haskell Embedded DSL(eDSL) for dataflow programming: CircuitFlow. Defined as a Symmetric Monoidal Preorder (SMP) on data that models dependencies in theworkflow, it has a strong mathematical basis, refocusing on how dataflows through an application, resulting in a more expressive solutionthat not only catches errors statically, but also achieves competitive runtime performance. In our preliminary evaluation, CircuitFlow outperformsthe industry-leading Luigi library of Spotify by scaling better with thenumber of inputs. The innovative creation of CircuitFlow is also of note,exemplifying how to create a modular eDSL whose semantics necessitates effects, and where storing complex type information for programcorrectness is paramount.

AB - Dataflow applications, such as machine learning algorithms,can run for days, making it desirable to have assurances that they willwork correctly. Current tools are not good enough: too often the interactions between tasks are not type-safe, leading to undesirable runtimeerrors. This paper presents a new declarative Haskell Embedded DSL(eDSL) for dataflow programming: CircuitFlow. Defined as a Symmetric Monoidal Preorder (SMP) on data that models dependencies in theworkflow, it has a strong mathematical basis, refocusing on how dataflows through an application, resulting in a more expressive solutionthat not only catches errors statically, but also achieves competitive runtime performance. In our preliminary evaluation, CircuitFlow outperformsthe industry-leading Luigi library of Spotify by scaling better with thenumber of inputs. The innovative creation of CircuitFlow is also of note,exemplifying how to create a modular eDSL whose semantics necessitates effects, and where storing complex type information for programcorrectness is paramount.

KW - eDSL

KW - Domain-specific languages

KW - Haskell

KW - Dataflow programming

U2 - 10.1007/978-3-030-94479-7_6

DO - 10.1007/978-3-030-94479-7_6

M3 - Conference Contribution (Conference Proceeding)

SN - 978-3-030-94478-0

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 79

EP - 98

BT - Practical Aspects of Declarative Languages

A2 - Cheney, James

A2 - Perri, Simona

PB - Springer Nature

ER -

Frohlich S , Wang M, Evans R. CircuitFlow: A Domain Specific Language for Dataflow Programming. In Cheney J, Perri S, editors, Practical Aspects of Declarative Languages: 24th International Symposium, PADL 2022, Philadelphia, PA, USA, January 17–18, 2022, Proceedings. Springer Nature. 2022. p. 79-98. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-94479-7_6

CircuitFlow: A Domain Specific Language for Dataflow Programming

Abstract

Publication series

Bibliographical note

Research Groups and Themes

Keywords

Access to Document

Handle.net

Fingerprint

Projects

8030 EPSRC EP/T008911/1 EXHIBIT : Expressive High-Level Languages for Bidirectional Transformations

Cite this