Flow System

[sergiimk]

About

Users must have an ability to trigger, schedule, monitor, and diagnose data processing tasks in the platform UI.

Product Requirements

As our platform ingests and processes data users need to be able to:

• See a list of currently running, scheduled, and past flows associated with a dataset
• See completion statuses of flows (success / failed)
• View flows logs to debug failures (POSTPONED)
• Trigger flows manually (“refresh” button?)
• Scheduling root dataset updates (cron-like?)
• Cancel scheduled or already running running flows

Tasks

Backend

• Introduce Admin users kamu-cli#276
• Prototype internal Event Bus (pub-sub) kamu-cli#279
• Flows: introduce a component to store dataset flows schedule settings kamu-cli#280
• Flows: support Flow History feature kamu-cli#282
• Flows: provide GraphQL API to schedule manual dataset flow kamu-cli#283
• Flows: implement automatic flow scheduler based on the user settings kamu-cli#284
• kamu ui: new flag to allow/disallow changing automatic dataset updates schedule kamu-cli#285
• Flows: Extend DatasetPermissions GraphQL type with canSchedule flag kamu-cli#287
• Flows: Fully support CRON expressions in the Flows System kamu-cli#396
• Flows: Design and implement GraphQL API to support Dataset Flows UI view features kamu-cli#397
• Flows: Provide GraphQL API to manually cancel flow's schedule tasks kamu-cli#398
• Flows: properly support batching/throttling conditions kamu-cli#401
• CRON expressions for flow scheduling should not have "seconds" parts kamu-cli#459
• Flows: improve queuing criteria for datasets kamu-cli#468
• Flows: provide GraphQL API that pauses all flows related to the current dataset kamu-cli#474
• Flows: provide API for server-side filtering of flows that respects pagination as well kamu-cli#476
• Flows: implement returning actual counters describing completed work in flow description API kamu-cli#495
• Flows: simplify waiting statuses and replace the differentiation on extended start conditions kamu-cli#516

Frontend

• Markup of key pages:
  • Dataset Flows page #233
  • Markup: Task Details View #234:
    • Markup: Task History View #235
• Implementing FE pages and integrating with the working GraphQL APIs:
  • Dataset Scheduling Settings #239
  • Flow paused and resume functionality for dataset updates #261
  • Implementation flow details/history tab #268
  • Add support batching/throttling conditions from UI #272
  • Add support for new start conditions and changes in flow statuses #275
  • Integrate cancellation of scheduled tasks for flow #277
  • Flows: Implement Summary tab in Flow Details view #279
• Additional tasks:
  • Introduce Admin users #232
  • Implement Refresh Now button on Overview page #240