Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingest flow does not honor the has_more flag #161

Open
sergiimk opened this issue Dec 12, 2024 · 0 comments
Open

Ingest flow does not honor the has_more flag #161

sergiimk opened this issue Dec 12, 2024 · 0 comments

Comments

@sergiimk
Copy link
Member

Consider com.defillama.tokens.prices dataset.

Its container will fetch data in 100 day intervals by default and signal has_more: true to kamu, so that pulling it in CLI results in multiple blocks committed as it pulls container until it's exhausted:
image

When I add this dataset to kamu node and run the ingest flow, however, it only executes one iteration and produces one AddData block. Thus I have to repeatedly trigger the flow until the source is exhausted.

Expectation:

  • Ingest flow should loop until the source is exhausted has_more == true
  • In the short term this looping could be done inside the ingest task, but ideally we would want each iteration to be in a separate task so that each of them went through the scheduling process and quotas checks independently
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant