Add models for per-entry & per-property metadata (#2167)

* Addded meta field to entry model * Added validator for meta field. * Update pyyaml version in requirements.txt * Update requirements.txt * Update requirements.txt * Remove test structures to get an idea of what triggers yaml import error on github. * Readding meta field to test_good_structures. * Readding meta field to test_structures.json. * Added handling None for property_metadata to validator + small correctionds. * Added test for validator per entry meta field. * Added test for presence metadata field in test_structures.py. * Remove metadata fields when the fields that they belong to are not returned. * add extra test for bad prefix. * Test if dependancy conflict causes error. * Revert "Test if dependancy conflict causes error." This reverts commit 6251e48. * Revert "Revert "Test if dependancy conflict causes error."" This reverts commit bf9692a. * correct version httpx. * commenting out validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue Github. * Slowly reassembling validator to see if this resolves issue Github. * Slowly reassembling validator to see if this resolves issue Github.. * Slowly reassembling validator to see if this resolves issue Github.. * Placed yaml import in try except block. * Added more cases to test data and added bugfix for removing associated metadata when a field is excluded. * Removed seemingly unneccesary mypy ignore exception statement. * remove change in version pyyaml in requirements.txt. * correct spelling mistake * moved starts_with_supported_prefix and check_starts_with_supported_prefix functions to BaseresouceMapper. * Expanded docstring check_starts_with_supported_prefix * Expanded docstring check_starts_with_supported_prefix * Added return type for starts_with_supported_prefix and check_starts_with_supported_prefix. * Added return type for starts_with_supported_prefix and check_starts_with_supported_prefix. * Adjusted validators baseed on suggestion Matthew. Co-authored-by: ml-evs [email protected] * small corrections validators. * small correction in removing unrequested metadata. * Added spaces before values 'exmpl_originates_from _project'. * update openapijson * update openapijson * update openapijson * use lstrip instead of manually removing / one at a time. * Update optimade/models/entries.py update description property_metadata field Co-authored-by: Matthew Evans <[email protected]> * Update description per entry meta datafield. * update openapijson. * try to see if moving yaml is still necessary. * Revert "try to see if moving yaml is still necessary." This reverts commit 68f14d3. * Added supported_prefixes field to config.py * Update requirements-client.txt (#1813) removed duplicate requirement * Modernize all Python 3.8 annotations (#1815) * Use Python 3.9 as the 'base' CI version for linting * Update pre-commit hooks * Run `pyupgrade --py39-plus` to upgrade legacy annotations * Add `--exit-non-zero-on-fix` for ruff Co-authored-by: Casper Welzel Andersen <[email protected]> * Use f-string over format Co-authored-by: Casper Welzel Andersen <[email protected]> --------- Co-authored-by: Casper Welzel Andersen <[email protected]> * Update pydantic validators and model configs for 2024 * Remove validation of 'supported prefix' in lieu of just checking leading underscores --------- Co-authored-by: Johan Bergsma <[email protected]> Co-authored-by: Casper Welzel Andersen <[email protected]>
Materials-Consortia · Oct 20, 2024 · 469a565 · 469a565
1 parent 58fc491
commit 469a565
Show file tree

Hide file tree

Showing 10 changed files with 257 additions and 27 deletions.
diff --git a/openapi/index_openapi.json b/openapi/index_openapi.json
@@ -477,6 +477,19 @@
         "title": "BaseRelationshipResource",
         "description": "Minimum requirements to represent a relationship resource"
       },
+      "EntryMetadata": {
+        "properties": {
+          "property_metadata": {
+            "type": "object",
+            "title": "Property Metadata",
+            "description": "An object containing per-entry and per-property metadata. The keys are the names of the fields in attributes for which metadata is available. The values belonging to these keys are dictionaries containing the relevant metadata fields. See also [Metadata properties](https://github.com/Materials-Consortia/OPTIMADE/blob/develop/optimade.rst#metadata-properties)"
+          }
+        },
+        "additionalProperties": true,
+        "type": "object",
+        "title": "EntryMetadata",
+        "description": "Contains the metadata for the attributes of an entry"
+      },
       "EntryRelationships": {
         "properties": {
           "references": {
@@ -536,13 +549,13 @@
           "meta": {
             "anyOf": [
               {
-                "$ref": "#/components/schemas/Meta"
+                "$ref": "#/components/schemas/EntryMetadata"
               },
               {
                 "type": "null"
               }
             ],
-            "description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
+            "description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
           },
           "attributes": {
             "$ref": "#/components/schemas/EntryResourceAttributes",
@@ -1283,13 +1296,13 @@
           "meta": {
             "anyOf": [
               {
-                "$ref": "#/components/schemas/Meta"
+                "$ref": "#/components/schemas/EntryMetadata"
               },
               {
                 "type": "null"
               }
             ],
-            "description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
+            "description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
           },
           "attributes": {
             "$ref": "#/components/schemas/LinksResourceAttributes",

diff --git a/openapi/openapi.json b/openapi/openapi.json
@@ -1797,6 +1797,19 @@
         ],
         "title": "EntryInfoResponse"
       },
+      "EntryMetadata": {
+        "properties": {
+          "property_metadata": {
+            "type": "object",
+            "title": "Property Metadata",
+            "description": "An object containing per-entry and per-property metadata. The keys are the names of the fields in attributes for which metadata is available. The values belonging to these keys are dictionaries containing the relevant metadata fields. See also [Metadata properties](https://github.com/Materials-Consortia/OPTIMADE/blob/develop/optimade.rst#metadata-properties)"
+          }
+        },
+        "additionalProperties": true,
+        "type": "object",
+        "title": "EntryMetadata",
+        "description": "Contains the metadata for the attributes of an entry"
+      },
       "EntryRelationships": {
         "properties": {
           "references": {
@@ -1856,13 +1869,13 @@
           "meta": {
             "anyOf": [
               {
-                "$ref": "#/components/schemas/Meta"
+                "$ref": "#/components/schemas/EntryMetadata"
               },
               {
                 "type": "null"
               }
             ],
-            "description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
+            "description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
           },
           "attributes": {
             "$ref": "#/components/schemas/EntryResourceAttributes",
@@ -2443,13 +2456,13 @@
           "meta": {
             "anyOf": [
               {
-                "$ref": "#/components/schemas/Meta"
+                "$ref": "#/components/schemas/EntryMetadata"
               },
               {
                 "type": "null"
               }
             ],
-            "description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
+            "description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
           },
           "attributes": {
             "$ref": "#/components/schemas/LinksResourceAttributes",
@@ -2943,13 +2956,13 @@
           "meta": {
             "anyOf": [
               {
-                "$ref": "#/components/schemas/Meta"
+                "$ref": "#/components/schemas/EntryMetadata"
               },
               {
                 "type": "null"
               }
             ],
-            "description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
+            "description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
           },
           "attributes": {
             "$ref": "#/components/schemas/ReferenceResourceAttributes"
@@ -4070,13 +4083,13 @@
           "meta": {
             "anyOf": [
               {
-                "$ref": "#/components/schemas/Meta"
+                "$ref": "#/components/schemas/EntryMetadata"
               },
               {
                 "type": "null"
               }
             ],
-            "description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
+            "description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
           },
           "attributes": {
             "$ref": "#/components/schemas/StructureResourceAttributes"

diff --git a/optimade/models/entries.py b/optimade/models/entries.py
@@ -1,9 +1,9 @@
 from datetime import datetime
 from typing import Annotated, Any, ClassVar, Literal
 
-from pydantic import BaseModel, field_validator
+from pydantic import BaseModel, ValidationInfo, field_validator
 
-from optimade.models.jsonapi import Attributes, Relationships, Resource
+from optimade.models.jsonapi import Attributes, Meta, Relationships, Resource
 from optimade.models.optimade_json import (
     BaseRelationshipResource,
     DataType,
@@ -117,6 +117,36 @@ def cast_immutable_id_to_str(cls, value: Any) -> str:
         return value
 
 
+class EntryMetadata(Meta):
+    """Contains the metadata for the attributes of an entry"""
+
+    property_metadata: dict = StrictField(
+        None,
+        description="""An object containing per-entry and per-property metadata. The keys are the names of the fields in attributes for which metadata is available. The values belonging to these keys are dictionaries containing the relevant metadata fields. See also [Metadata properties](https://github.com/Materials-Consortia/OPTIMADE/blob/develop/optimade.rst#metadata-properties)""",
+    )
+
+    @field_validator("property_metadata", mode="before")
+    def check_property_metadata_subfields(cls, value: Any) -> dict:
+        """Loop through any per-property metadata field and check that
+        the subfields are prefixed correctly.
+
+        """
+        error_fields: list[str] = []
+
+        if value is not None and isinstance(value, dict):
+            for field in value:
+                if property_metadata := value.get(field):
+                    for subfield in property_metadata:
+                        if not subfield.startswith("_"):
+                            error_fields.append(subfield)
+
+        if error_fields:
+            raise ValueError(
+                f"The keys under the field `property_metadata` need to be prefixed. The field(s) {error_fields} are not prefixed."
+            )
+        return value
+
+
 class EntryResource(Resource):
     """The base model for an entry resource."""
 
@@ -171,6 +201,14 @@ class EntryResource(Resource):
         ),
     ]
 
+    meta: Annotated[
+        EntryMetadata | None,
+        StrictField(
+            None,
+            description="""A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata.""",
+        ),
+    ] = None
+
     relationships: Annotated[
         EntryRelationships | None,
         StrictField(
@@ -179,6 +217,44 @@ class EntryResource(Resource):
         ),
     ] = None
 
+    @field_validator("meta", mode="before")
+    def check_meta(cls, meta: Any, info: ValidationInfo) -> dict | None:
+        """Validator to check whether the per-entry `meta` field is valid,
+        including stripping out any per-property metadata for properties that
+        do not otherwise appear in the model.
+
+        """
+        if not meta:
+            return meta
+
+        if property_metadata := meta.pop("property_metadata", None):
+            # check that all the fields under property metadata are in attributes
+            attributes = info.data.get("attributes", {})
+            property_error_fields: list[str] = []
+            for subfield in property_metadata:
+                if subfield not in attributes:
+                    property_error_fields.append(subfield)
+
+            if property_error_fields:
+                raise ValueError(
+                    f"The keys under the field `property_metadata` need to match with the field names in attributes. The field(s) {property_error_fields} are however not present in attributes."
+                )
+
+        meta["property_metadata"] = property_metadata
+
+        meta_error_fields: list[str] = []
+        for field in meta:
+            if field not in EntryMetadata.model_fields:
+                if not field.startswith("_"):
+                    meta_error_fields.append(field)
+
+        if meta_error_fields:
+            raise ValueError(
+                f"The keys under the field `meta` need to be prefixed if not otherwise defined. The field(s) {meta_error_fields} are not defined for per-entry `meta`."
+            )
+
+        return meta
+
 
 class EntryInfoProperty(BaseModel):
     description: Annotated[

diff --git a/optimade/server/data/test_structures.json b/optimade/server/data/test_structures.json
@@ -3,6 +3,13 @@
     "_id": {
       "$oid": "5cfb441f053b174410700d02"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {
+          "_exmpl_originates_from_project": "Pure Metals"
+        }
+      }
+    },
     "assemblies": null,
     "chemsys": "Ac",
     "cartesian_site_positions": [
@@ -80,6 +87,13 @@
     "_id": {
       "$oid": "5cfb441f053b174410700d03"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {
+          "_exmpl_originates_from_project": "Actinides_Alloys"
+        }
+      }
+    },
     "assemblies": null,
     "chemsys": "Ac-Ag-Ir",
     "cartesian_site_positions": [
@@ -197,6 +211,13 @@
     "_id": {
       "$oid": "5cfb441f053b174410700d04"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {
+          "_exmpl_originates_from_project": "Actinides_Alloys"
+        }
+      }
+    },
     "assemblies": null,
     "chemsys": "Ac-Ag-Pb",
     "cartesian_site_positions": [
@@ -323,6 +344,13 @@
     "_id": {
       "$oid": "5cfb441f053b174410700d18"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {
+          "_exmpl_originates_from_project": "Actinides_Alloys"
+        }
+      }
+    },
     "assemblies": null,
     "chemsys": "Ac-Mg",
     "cartesian_site_positions": [
@@ -413,6 +441,13 @@
     "_id": {
       "$oid": "5cfb441f053b174410700d1f"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {
+          "_exmpl_originates_from_project": null
+        }
+      }
+    },
     "assemblies": null,
     "chemsys": "Ac-O",
     "cartesian_site_positions": [
@@ -515,6 +550,11 @@
     "_id": {
       "$oid": "5cfb441f053b174410700d6f"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {}
+      }
+    },
     "assemblies": null,
     "chemsys": "Ac-Cu-F-O",
     "cartesian_site_positions": [
@@ -639,6 +679,13 @@
     "_id": {
       "$oid": "5cfb441f053b174410700dc9"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": {
+          "_exmpl_originates_from_project": "Pure Metals"
+        }
+      }
+    },
     "assemblies": null,
     "chemsys": "Ag",
     "cartesian_site_positions": [
@@ -706,6 +753,11 @@
     "_id": {
       "$oid": "5cfb441f053b174410700ddd"
     },
+    "meta": {
+      "property_metadata": {
+        "elements_ratios": null
+      }
+    },
     "assemblies": null,
     "chemsys": "Ag-Br-Cl-Te",
     "cartesian_site_positions": [
@@ -896,6 +948,9 @@
     "_id": {
       "$oid": "5cfb441f053b174410700e04"
     },
+    "meta": {
+      "property_metadata": {}
+    },
     "assemblies": null,
     "chemsys": "Ag-C-Cl-N-O-S",
     "cartesian_site_positions": [
@@ -1072,6 +1127,9 @@
     "_id": {
       "$oid": "5cfb441f053b174410700e11"
     },
+    "meta": {
+      "property_metadata": null
+    },
     "assemblies": null,
     "chemsys": "Ag-C-Cl-H-N",
     "cartesian_site_positions": [