OPTIMADE JSON Lines specification appendix #531

ml-evs · 2024-07-19T13:12:19Z

This PR begins the work on #471, by defining the conventions on top of JSON Lines for deserializing an entire OPTIMADE API into a file.

optimade.rst

Fix code block formatting Update optimade.rst Apply suggestions from code review Apply suggestions from code review Try to fix formatting again Fix formatting and add note about provider fields

eimrek

Thanks @ml-evs, and sorry for the delays. Now had some time to take a look. I like the proposal and it can be used for optimade-maker as well. But I wrote some comments for consideration.

eimrek · 2024-08-27T20:47:42Z

optimade.rst

+
+The `JSON Lines <https://jsonlines.org/>`__ format enforces the following rules:
+
+- each line is a valid JSON object,


Suggested change

- each line is a valid JSON object,

- each line is a valid JSON value,

Is this more precise?

Hmm, not sure, to me value implies something stored under a key, whereas object is the smallest chunk of data that can be parsed entirely as JSON?

on https://jsonlines.org/ they mention "values" (as even e.g. just strings or integers are supported). But I agree that in the vast majority of cases, people will use objects.

optimade.rst

eimrek · 2024-08-27T22:38:52Z

optimade.rst

+
+- The next line MAY contain a standard OPTIMADE ``meta`` object, following the same rules described in `JSON Response Schema: Common Fields`_, where every MUST and SHOULD rule can be reinterpreted as a MAY rule.
+- The next block of lines provides the ``info`` endpoint responses.
+  - First the base info response MUST be provided, following the description at `Base Info Endpoint`_.


MUST

This breaks current optimade-maker jsonl files (well, just the li-ion-conductors one). Would a MAY, in principle, work here?

This is quite important IMO, we don't provide any other way of identifying the file (and in fact we might want to even expand this with a version of the JSONL specification like {'x-optimade': {'api_version': 1.1, 'jsonl_version': 1.2} if we want the JSONL file to be able to store OPTIMADE data of any version. Would you be against simply adding it to the files where it is missing atm?

Sorry, not sure if my comment was precise enough. I was commenting on the base info endpoint, which doesn't contain any crucial info as far as i understand, but I see how it can be useful to know what follows in the next lines.

eimrek · 2024-08-27T22:39:29Z

optimade.rst

+- The next line MAY contain a standard OPTIMADE ``meta`` object, following the same rules described in `JSON Response Schema: Common Fields`_, where every MUST and SHOULD rule can be reinterpreted as a MAY rule.
+- The next block of lines provides the ``info`` endpoint responses.
+  - First the base info response MUST be provided, following the description at `Base Info Endpoint`_.
+  - The next lines MUST contain the entry info endpoint responses for the all entry types present later in the file, as described in `Entry Listing Info Endpoints`_. These MUST appear in alphabetical order by entry type name.


These MUST appear in alphabetical order by entry type name.

Fine for me, but is it needed?

Happy to relax this constraint, probably we'll never need to worry about an OPTIMADE JSONL file containing 100s of entry types (in which case you could use alphabetical order to short-circuit the search for a given entry type).

ok, i see, but anyway, fine for me. good to keep consistent if we require the entry blocks to be sorted, where this is more valuable, i think.

eimrek · 2024-08-27T22:44:38Z

optimade.rst

+  {"type": "structures", "id": "1", "attributes": {...}}
+  {"type": "references", "id": "2", "attributes": {...}}


references block should be ahead of structures, as required above. But again, is the alphabetical ordering needed?

Co-authored-by: Kristjan Eimre <[email protected]>

eimrek · 2024-08-28T19:47:47Z

optimade.rst

+
+- The next line MAY contain a standard OPTIMADE ``meta`` object, following the same rules described in `JSON Response Schema: Common Fields`_, where every MUST and SHOULD rule can be reinterpreted as a MAY rule.
+- The next block of lines provides the ``info`` endpoint responses.
+  - First the base info response MUST be provided, following the description at `Base Info Endpoint`_.


Sorry, not sure if my comment was precise enough. I was commenting on the base info endpoint, which doesn't contain any crucial info as far as i understand, but I see how it can be useful to know what follows in the next lines.

eimrek · 2024-08-28T19:51:11Z

optimade.rst

+- The next line MAY contain a standard OPTIMADE ``meta`` object, following the same rules described in `JSON Response Schema: Common Fields`_, where every MUST and SHOULD rule can be reinterpreted as a MAY rule.
+- The next block of lines provides the ``info`` endpoint responses.
+  - First the base info response MUST be provided, following the description at `Base Info Endpoint`_.
+  - The next lines MUST contain the entry info endpoint responses for the all entry types present later in the file, as described in `Entry Listing Info Endpoints`_. These MUST appear in alphabetical order by entry type name.


ok, i see, but anyway, fine for me. good to keep consistent if we require the entry blocks to be sorted, where this is more valuable, i think.

eimrek · 2024-08-28T19:51:35Z

optimade.rst

+  {"type": "structures", "id": "1", "attributes": {...}}
+  {"type": "references", "id": "2", "attributes": {...}}


Suggested change

{"type": "structures", "id": "1", "attributes": {...}}

{"type": "references", "id": "2", "attributes": {...}}

{"type": "references", "id": "1", "attributes": {...}}

{"type": "structures", "id": "2", "attributes": {...}}

switched em