Data analysis workflows (DAWs) are widely used in the scientific world. However, different communities created a plethora of domain-specific languages (DSLs) to specify their DAWs. Consequently, across DSLs, it is hard to perform operations on DAWs such as share, port, compare, re-use, adapt, or even merge. Thus, we have analyzed DAW specification languages and created a unified DAW metamodel to better understand the different DAW specification languages.

Scientific Workflow Management Systems (SWMS) are software systems designed to enable the scalable, distributed, and reproducible execution of complex data analysis workflows on large datasets. Due to the importance of such analyses, a plethora of different systems have been built over the last decades. Although all of them share the same core functions of allowing workflow specification, controlling task dependencies, and steering the correct execution of workflows on a given computational infrastructure, they differ notably in the specific implementations of these functionalities and often offer additional features that can benefit workflow developers. The differences are often subtle, yet impactful, and often lack proper documentation leading to unpleasant surprises when trying to port a workflow developed for one SWMS to another or when re-implementing a stand-alone application with an SWMS. In this chapter, we want to highlight some of the main differences between workflow systems by comparing the properties and features of four SWMSs: Nextflow, Airflow, Argo, and Snakemake. The comparison is conducted using SWMSs to reimplement a complex workflow from the remote sensing domain that analyzes satellite images to study the development of vegetation over the years on the island of Crete. We find and describe important distinctions between these systems in numerous aspects, including file handling, scheduling, parallelization strategies, and language elements. We believe that awareness of these differences and the difficulties they might incur in a specific setting is important for making informed decisions when choosing an SWMS for a new research project.

Modern workflow systems typically rely on static resource allocation that does not change during the execution and is often based on rough user resource estimates. As a result, resource allocation and scheduling may be underprovisioning, which leads to slowing down the execution, or overprovisioning, which causes wasting valuable resources.
Leveraging detailed knowledge regarding a task’s behavior, we can optimize runtime, resource consumption, and execution reliability. For that purpose, an innovative approach to modeling tasks, workflows, and their execution is used as part of the scheduling. Contrary to the typical workflow execution, this modeling is done on sub-task granularity to be able to proactively allocate parts of the resources just in time.
Since the accuracy of the model and the resulting predictions cannot be free of errors/noise, the decision-making for resource allocation and scheduling is repeated continuously during execution using updated execution metrics from monitoring data of the task. After the execution of a task, monitoring data from executions is used to refine the model and reduce the error in future predictions. This, in turn, enables a more efficient execution of the task in the current workflow execution and potentially for other workflows running on different infrastructures using this task.
This chapter presents a comprehensive workflow optimization approach with a focus on data intensive workflows. It integrates past work by the authors on specific aspects of the methodology with novel ideas that will be explored further in future research.

The amount of satellite data in Earth Observation (EO) is rapidly increasing, providing scientists with new opportunities to study how climate, weather events, and direct anthropogenic factors affect the Earth’s surface. However, the urgency for more accurate and holistic studies drives growth in the size of analyzed data sets (i.e., higher spatial resolution, larger study areas, increasing temporal depth) and complexity of analysis, leading to more complex and more data-intensive workflows. Analyzing such large data sets requires distributed computing resources, which are hard to program and often require specialized expertise to achieve satisfying runtimes. As a result, Earth Observation data to-date are often underused.
Recently, scientific workflow management systems (SWMS) emerged as a new programming paradigm for complex analysis pipelines over large data sets executed on distributed compute resources. They promise simple development, improved portability across systems, automatic scalability on different infrastructures, easier reuse of workflows, and reproducibility of analysis results. As such, using SWMS for EO analysis can boost the more efficient use of EO data and facilitate the dissemination of standardized data pre-processing and processing pipelines.
In this book chapter, we delve into the application of SWMS for Earth Observation. Specifically, we describe three research projects in which we used Nextflow, a popular open source scientific workflow engine, for programming portable and scalable data analysis pipelines. To this end, we describe SWMS in general and specifically Nextflow regarding their suitability for EO data analysis, and give practical examples to highlight advantages and challenges when using SWMS for analyzing large sets of satellite images.

Scientific workflow management systems support large-scale data analysis on cluster infrastructures. For this, they interact with resource managers which schedule workflow tasks onto cluster nodes. In addition to workflow task descriptions, resource managers rely on task performance estimates such as main memory consumption and runtime to efficiently manage cluster resources. Such performance estimates should be automated, as user-based task performance estimates are error-prone.
In this book chapter, we describe key characteristics of methods for workflow task runtime and memory prediction, provide an overview and a detailed comparison of state-of-the-art methods from the literature, and discuss how workflow task performance prediction is useful for scheduling, energy-efficient and carbon-aware computing, and cost prediction.