Streamflow (software)
Arthurseat (talk) 23:06, 27 August 2022 (UTC)
Search Streamflow (software) on Amazon.
StreamFlow is an open-source workflow management platform for data engineering pipelines. It started at University of Turin in 2021[1] as an implementation of the Common Workflow Language (CWL) open standard aiming at supporting the portability of workflow code across different platforms, such as High-performance computing, private/public clouds, and hybrid cloud-HPC infrastructures. Tools and workflows described using CWL are portable across various platforms that support the CWL standards.
StreamFlow is a container-native WMS written in Python. StreamFlow workflows (a.k.a. pipelines) are CWL documents[2], which are written in YAML (and/or JSON). StreamFlow is designed under Dijkstra's principle of "separation of concerns" [3]. A workflow is defined by describing two concerns: 1) the data dependencies among workflow steps, usually represented by files, and 2) the mapping of workflow steps onto infrastructures, mediated by their workload managers, e.g. SLURM, PBS, etc.), cloud orchestrator APIs (Kubernetes, etc.), or direct access to the UNIX shell (via ssh and public key). The separation between data dependencies and the mapping enables portability since moving to a different execution infrastructure only requires the revision of the mapping document.
CWL workflows describe each step with explicit inputs and outputs. Workflow steps in CWL are not necessarily run in the order they are listed. Instead, the order is determined by the dependencies between steps. Workflow steps that do not depend on one another may run in parallel. Additionally, the scatter feature in CWL allows the repeated execution of a CWL step in parallel (depending on the resources available) over a list of inputs. This can be done without requiring the modification of the underlying tool description.
StreamFlow supports the execution of workflow steps in multi-container distributed environments; it does not require any shared storage for different processing elements: data is automatically (and securely) moved between infrastructures running different workflow steps.
Adoption
The COVID CLAIRE task force[4] has adopted StreamFlow for developing early pipelines to detect COVID-19 pneumonia lesion categorization from CT-scans [5]. StreamFlow is adopted as background technology by large projects, such as the 8M€ EuroHPC "ACROSS project (2021-2024)".
References
- ↑
Colonnelli, Iacopo; Cantalupo, Barbara; Merelli, Ivan; Aldinucci, Marco (2021). "StreamFlow: cross-breeding cloud with HPC". IEEE Transactions on Emerging Topics in Computing. 9 (4): 1723–1737. arXiv:2002.01558. doi:10.1109/TETC.2020.3019202. Retrieved August 27, 2022. Unknown parameter
|s2cid=ignored (help) - ↑
Crusoe, Michael R.; Abeln, Sanne; Iosup, Alexandru; Amstutz, Peter; Chilton, John; Tijanić, Nebojša; Ménager, Hervé; Soiland-Reyes, Stian; Gavrilović, Bogdan; Goble, Carole (2022). "Methods included: standardizing computational reuse and portability with the Common Workflow Language". Communications of the ACM. 65 (6): 54–63. doi:10.1145/3486897. Retrieved August 27, 2022. Unknown parameter
|s2cid=ignored (help) - ↑
Dijkstra, Edsger W. (1974). "On the role of scientific thought" (PDF). Edsger W. Dijkstra Archive. Retrieved August 27, 2022. Unknown parameter
|url-status=ignored (help) - ↑
Bontempi, Gianluca (2021). "The CLAIRE COVID-19 initiative: approach, experiences and recommendations". Ethics and Information Technology. 23 (Suppl 1): 127–133. doi:10.1007/s10676-020-09567-7. PMC 7871022 Check
|pmc=value (help). PMID 33584129 Check|pmid=value (help). Unknown parameter|s2cid=ignored (help) - ↑
Pennisi, Matteo (2021). "An explainable AI system for automated COVID-19 assessment and lesion categorization from CT-scans". Artificial Intelligence in Medicine. 118: 102114. doi:10.1016/j.artmed.2021.102114. PMC 8139171 Check
|pmc=value (help). PMID 34412837 Check|pmid=value (help). Unknown parameter|s2cid=ignored (help)
External links
This article "Streamflow (software)" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:Streamflow (software). Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.
