Flows System round 2

[sergiimk]

About

Improve usability of flows system.

Product Requirements

• View flows logs to debug failures
• View lists of flows for all datasets of the user
• Integrate new types of flows: COMPACTING

Tasks

Backend:

• planned:

  • Flows: Integrate SQLX repository for persistent flow configurations kamu-cli#578
  • Flows: Integrate SQLX repository for persistent flow event store kamu-cli#598
  • Compacting: integrate into Flows system kamu-cli#583
  • Store and stream task execution logs kamu-cli#286
  • Flows: provide API to list flows from all datasets of the specified account kamu-cli#551
  • Flows: improve sorting of listed flows by event times kamu-cli#604

• out of stage scope:

  • GraphQL: support live updates of Flows when their state evolves kamu-cli#275
  • Authorization checks for Tasks GraphQL API kamu-cli#277 (needs to be reconsidered)
  • GraphQL: Administrators view of Executors kamu-cli#278
  • GraphQL: Administator's view of tasks kamu-cli#281
  • Flows: model simple scoring function to prioritize running flows in the same time slot kamu-cli#399
  • Flows: retry logic for failed tasks kamu-cli#400
  • Flows: Automatically pause flow schedules if dependent tasks are failing  kamu-cli#419

Frontend:

• planned:

  • Markup: Task Logs View kamu-web-ui#236
  • Flows: Ability to view summary of flows for all datasets of the account kamu-web-ui#280
  • Integrate Hard Compacting flow type into Flows System UI kamu-web-ui#302

• out of stage scope:

  • Real-time status updates for Flows kamu-web-ui#231
  • Markup: Administrator Tasks View kamu-web-ui#237
  • Markup: Administrator Executors View kamu-web-ui#238
  • e2e tests (TBD)