Ability to dynamically skip erroneous entries

[anyangml]

Hi,
I am trying to use the Optimade API to download the Aflow dataset, however, I believe there is a server-side issue at the following endpoint https://aflow.org/API/optimade/v1/structures?page_number=758&page_limit=100

The API will always crash at this point. I can manually loop through the pages with requests, but I wonder if there is an option to skip bad endpoints using the optimade client?

[ml-evs]

Hi @anyangml, you should be able to filter out providers in the CLI with

  --exclude-providers TEXT        A string of comma-separated provider IDs to
                                  exclude from queries.

e.g., optimade-get --exclude-providers aflow,oqmd,mp, or if you are using the client within your own script, then you can pass exclude_providers=["aflow", "oqmd", "mp", "whoever"] to the OptimadeClient on initialisation.

I think generally, the AFLOW team would appreciate it if you report issues too! I can confirm that I also get a server error with that link, and it seems like they now provide an email address and issue tracker in the JSON response that you could use to report the problem.

[anyangml]

Hi @anyangml, you should be able to filter out providers in the CLI with

  --exclude-providers TEXT        A string of comma-separated provider IDs to
                                  exclude from queries.

e.g., optimade-get --exclude-providers aflow,oqmd,mp, or if you are using the client within your own script, then you can pass exclude_providers=["aflow", "oqmd", "mp", "whoever"] to the OptimadeClient on initialisation.

I think generally, the AFLOW team would appreciate it if you report issues too! I can confirm that I also get a server error with that link, and it seems like they now provide an email address and issue tracker in the JSON response that you could use to report the problem.

Oh, I didn't mean to skip the entire provider. I just want to skip the bad endpoints within Aflow. I believe this can be done with requests if I loop through the page numbers.

[ml-evs]

Ah I see, that's not currently possible though we could add it as an option (and e.g., could automatically switch to crawling entry by entry to find the bad one).

Can you verify that the pages afterwards actually work though? Obviously it would be better if AFLOW could just fix this page for you.

I'd be happy to accept a PR for this, otherwise I will add it to the backlog.

[anyangml]

Ah I see, that's not currently possible though we could add it as an option (and e.g., could automatically switch to crawling entry by entry to find the bad one).

Can you verify that the pages afterwards actually work though? Obviously it would be better if AFLOW could just fix this page for you.

I'd be happy to accept a PR for this, otherwise I will add it to the backlog.

I think there are similar issues in the following pages and I agree that AFLOW should look into this issue. I have reported this issue to Aflow via email, and I may create a PR to add a skip option when I have time (No guarantee though).