Use case: exporting a provenance graph of workflow execution as RO-crate

[sphuber]

Since I am only still familiarizing myself with RO-crate and I am not even sure that it is the right solution for what I am trying to do, I decided to open this as a discussion instead of an issue. Feel free to convert if that is more appropriate.

I am involved in the AiiDA project which is an open-source workflow management system with a strong focus on provenance. When a workflow is executed in AiiDA, its execution is recorded in a directed graph, where the nodes are the workflow steps and the various inputs and outputs that they produce. AiiDA provides a way to export (part) of this provenance graph in an archive file, which can then be shared and imported into another installation. However, this is a custom format and only interpretable by AiiDA. We are now investigating if we could export it into a more standardized format, in this RO-crate, such that it would be somewhat intelligible even outside the scope of AiiDA.

The main problem that I see so far is that the AiiDA entities that I need to describe in the RO-crate, do not really match any of the types that I see described in the documentation. To give a simple example, consider the following minimal graph, typical for AiiDA:

This is a very simple workflow that consists of a single step (calculation) which is represented by the magenta square. The step consumed three inputs (of various types) and produced two outputs, all represented by the green circles. Each node (square, or circle) contains metadata (JSON-serializable) and optionally one or more files.

How, if at all possible, would I describe this in a RO-crate? Should each node be represented by a DataEntity where the metadata is written as a JSON file, and all other files are linked through the hasPart key? Is there a way that I can "formalize" the link between these nodes?

[simleo]

Hi Sebastiaan. How to represent workflow run provenance in RO-Crate is still an open problem. At the moment, the RO-Crate spec contains only high-level guidance on provenance, but we are working on a set of profiles for describing the provenance of a workflow execution in a more detailed way. Recently, we formed a working group, with regular meetings every two weeks, to bring this forward together with several interested parties. The next meeting is this afternoon at 4 PM CET. It would be great if you could join the meeting (or maybe one of the future ones, if you can't make it today) and present your use case so we can discuss it together. You are also very welcome to join the working group, of course, which you can do by posting here. If you join the ro-crate Slack channel we'll be able to give you more details. Thanks for reaching out!