Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce tasks which produce intermediate results #126

Merged
merged 14 commits into from
Mar 8, 2024
Merged

Conversation

WardLT
Copy link
Collaborator

@WardLT WardLT commented Mar 7, 2024

Overhaul how tasks are defined in Colmena to provide the ability to distinguish tasks which produce intermediate results from those that do not.

Previously, we wrapped functions being invoked by a workflow engine in a function which handled the Colmena-specific things like unpacking results.

The new approach is simply a class-based strategy for wrapping. The logic which was in the wrapper function is now in the __call__ method of the class, and the function being invoked is a attribute of the class.

The class based approach gives more opportunities for creating subclasses, like the new PythonGeneratorTask and the legacy ExecutableTask class, which invoke the user-provided function differently.

To do list:

  • Write unit tests for new models
  • Update task servers to use new models
  • Add a new section to the documentation about task types
  • Unbreak the MPI example

@WardLT WardLT marked this pull request as ready for review March 8, 2024 14:18
@WardLT WardLT merged commit 3c20721 into master Mar 8, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant