Add Postgres JSON filters

docs/query.rst

@@ -4,45 +4,45 @@
 Query API
 =========
 
-This document describes how to use QuerySet to build your queries
+This document describes how to use QuerySet to query the database.
 
-Be sure to check `examples <https://github.com/tortoise/tortoise-orm/tree/master/examples>`_ for better understanding
+Be sure to check `examples <https://github.com/tortoise/tortoise-orm/tree/master/examples>`_.
 
-You start your query from your model class:
+Below is an example of a simple query that will return all events with a rating greater than 5:
 
 .. code-block:: python3
 
-    Event.filter(id=1)
+    await Event.filter(rating__gt=5)
 
 There are several method on model itself to start query:
 
 - ``filter(*args, **kwargs)`` - create QuerySet with given filters
 - ``exclude(*args, **kwargs)`` - create QuerySet with given excluding filters
 - ``all()`` - create QuerySet without filters
-- ``first()`` - create QuerySet limited to one object and returning instance instead of list
+- ``first()`` - create QuerySet limited to one object and returning the first object
 - ``annotate()`` - create QuerySet with given annotation
 
-This method returns ``QuerySet`` object, that allows further filtering and some more complex operations
+The methods above return a ``QuerySet`` object, which supports chaining query operations.
 
-Also model class have this methods to create object:
+The following methods can be used to create an object:
 
-- ``create(**kwargs)`` - creates object with given kwargs
-- ``get_or_create(defaults, **kwargs)`` - gets object for given kwargs, if not found create it with additional kwargs from defaults dict
+- ``create(**kwargs)`` - creates an object with given kwargs
+- ``get_or_create(defaults, **kwargs)`` - gets an object for given kwargs, if not found create it with additional kwargs from defaults dict
 
-Also instance of model itself has these methods:
+The instance of a model has the following methods:
 
 - ``save()`` - update instance, or insert it, if it was never saved before
 - ``delete()`` - delete instance from db
 - ``fetch_related(*args)`` - fetches objects related to instance. It can fetch FK relation, Backward-FK relations and M2M relations. It also can fetch variable depth of related objects like this: ``await team.fetch_related('events__tournament')`` - this will fetch all events for team, and for each of this events their tournament will be prefetched too. After fetching objects they should be available normally like this: ``team.events[0].tournament.name``
 
-Another approach to work with related objects on instance is to query them explicitly in ``async for``:
+Another approach to work with related objects on instance is to query them explicitly with ``async for``:
 
 .. code-block:: python3
 
     async for team in event.participants:
         print(team.name)
 
-You also can filter related objects like this:
+The related objects can be filtered:
 
 .. code-block:: python3
 
@@ -53,7 +53,7 @@ which will return you a QuerySet object with predefined filter
 QuerySet
 ========
 
-After you obtained queryset from object you can do following operations with it:
+Once you have a QuerySet, you can perform the following operations with it:
 
 .. automodule:: tortoise.queryset
     :members:
@@ -64,8 +64,8 @@ After you obtained queryset from object you can do following operations with it:
     .. autoclass:: QuerySet
         :inherited-members:
 
-QuerySet could be constructed, filtered and passed around without actually hitting database.
-Only after you ``await`` QuerySet, it will generate query and run it against database.
+QuerySet could be constructed, filtered and passed around without actually hitting the database.
+Only after you ``await`` QuerySet, it will execute the query.
 
 Here are some common usage scenarios with QuerySet (we are using models defined in :ref:`getting_started`):
 
@@ -113,7 +113,7 @@ You could do it using ``.prefetch_related()``:
     # This will fetch tournament with their events and teams for each event
     tournament_list = await Tournament.all().prefetch_related('events__participants')
 
-    # Fetched result for m2m and backward fk relations are stored in list-like container
+    # Fetched result for m2m and backward fk relations are stored in list-like containe#r
     for tournament in tournament_list:
         print([e.name for e in tournament.events])
 
@@ -194,12 +194,15 @@ You can use them like this:
 Filtering
 =========
 
-When using ``.filter()`` method you can use number of modifiers to field names to specify desired operation
+When using the ``.filter()`` method, you can apply various modifiers to field names to specify the desired lookup type.
+In the following example, we filter the Team model to find all teams whose names contain the string CON (case-insensitive):
 
 .. code-block:: python3
 
     teams = await Team.filter(name__icontains='CON')
 
+The following lookup types are available:
+
 - ``not``
 - ``in`` - checks if value of field is in passed list
 - ``not_in``
@@ -219,26 +222,21 @@ When using ``.filter()`` method you can use number of modifiers to field names t
 - ``iexact`` - case insensitive equals
 - ``search`` - full text search
 
-Specially, you can filter date part with one of following, note that current only support PostgreSQL and MySQL, but not sqlite:
-
-.. code-block:: python3
+For PostgreSQL and MySQL, the following date related lookup types are available:
 
-    class DatePart(Enum):
-        year = "YEAR"
-        quarter = "QUARTER"
-        month = "MONTH"
-        week = "WEEK"
-        day = "DAY"
-        hour = "HOUR"
-        minute = "MINUTE"
-        second = "SECOND"
-        microsecond = "MICROSECOND"
+- ``year`` - e.g. ``await Team.filter(created_at__year=2020)``
+- ``quarter``
+- ``month``
+- ``week``
+- ``day``
+- ``hour``
+- ``minute``
+- ``second``
+- ``microsecond``
 
-    teams = await Team.filter(created_at__year=2020)
-    teams = await Team.filter(created_at__month=12)
-    teams = await Team.filter(created_at__day=5)
 
-In PostgreSQL and MYSQL, you can use the ``contains``, ``contained_by`` and ``filter`` options in ``JSONField``:
+In PostgreSQL and MYSQL, you can use the ``contains``, ``contained_by`` and ``filter`` options in ``JSONField``.
+The ``filter`` option allows you to filter the JSON object by its keys and values.
 
 .. code-block:: python3
 
@@ -254,11 +252,6 @@ In PostgreSQL and MYSQL, you can use the ``contains``, ``contained_by`` and ``fi
 
     objects = await JSONModel.filter(data__contained_by=["text", "tortoise", "msg"])
 
-.. code-block:: python3
-
-    class JSONModel:
-        data = fields.JSONField[dict]()
-
     await JSONModel.create(data={"breed": "labrador",
                                  "owner": {
                                      "name": "Boby",
@@ -279,7 +272,8 @@ In PostgreSQL and MYSQL, you can use the ``contains``, ``contained_by`` and ``fi
     obj6 = await JSONModel.filter(data__filter={"owner__last__not_isnull": False}).first()
 
 In PostgreSQL and MySQL, you can use ``postgres_posix_regex`` to make comparisons using POSIX regular expressions:
-On PostgreSQL, this uses the ``~`` operator, on MySQL it uses the ``REGEXP`` operator.
+In PostgreSQL, this is done with the ``~`` operator, while in MySQL the ``REGEXP`` operator is used.
+
 
 .. code-block:: python3
     class DemoModel:
@@ -289,6 +283,23 @@ On PostgreSQL, this uses the ``~`` operator, on MySQL it uses the ``REGEXP`` ope
     obj = await DemoModel.filter(demo_text__posix_regex="^Hello World$").first()
 
 
+In PostgreSQL, ``filter`` supports additional lookup types:
+
+- ``in`` - ``await JSONModel.filter(data__filter={"breed__in": ["labrador", "poodle"]}).first()``
+- ``not_in``
+- ``gte``
+- ``gt``
+- ``lte``
+- ``lt``
+- ``range`` - ``await JSONModel.filter(data__filter={"age__range": [1, 10]}).first()``
+- ``startswith``
+- ``endswith``
+- ``iexact``
+- ``icontains``
+- ``istartswith``
+- ``iendswith``
+
+
 Complex prefetch
 ================
 

tests/contrib/postgres/test_json.py

@@ -0,0 +1,140 @@
+from datetime import datetime
+from decimal import Decimal
+
+from tests.testmodels import JSONFields
+from tortoise.contrib import test
+from tortoise.exceptions import DoesNotExist
+
+
+@test.requireCapability(dialect="postgres")
+class TestPostgresJSON(test.TestCase):
+    async def asyncSetUp(self):
+        await super().asyncSetUp()
+        self.obj = await JSONFields.create(
+            data={
+                "val": "word1",
+                "int_val": 123,
+                "float_val": 123.1,
+                "date_val": datetime(1970, 1, 1, 12, 36, 59, 123456),
+                "int_list": [1, 2, 3],
+                "nested": {
+                    "val": "word2",
+                    "int_val": 456,
+                    "int_list": [4, 5, 6],
+                    "date_val": datetime(1970, 1, 1, 12, 36, 59, 123456),
+                    "nested": {
+                        "val": "word3",
+                    },
+                },
+            }
+        )
+
+    async def get_by_data_filter(self, **kwargs) -> JSONFields:
+        return await JSONFields.get(data__filter=kwargs)
+
+    async def test_json_in(self):
+        self.assertEqual(await self.get_by_data_filter(val__in=["word1", "word2"]), self.obj)
+        self.assertEqual(await self.get_by_data_filter(val__not_in=["word3", "word4"]), self.obj)
+
+        with self.assertRaises(DoesNotExist):
+            await self.get_by_data_filter(val__in=["doesnotexist"])
+
+    async def test_json_defaults(self):
+        self.assertEqual(await self.get_by_data_filter(val__not="word2"), self.obj)
+        self.assertEqual(await self.get_by_data_filter(val__isnull=False), self.obj)
+        self.assertEqual(await self.get_by_data_filter(val__not_isnull=True), self.obj)
+
+    async def test_json_int_comparisons(self):
+        self.assertEqual(await self.get_by_data_filter(int_val=123), self.obj)
+        self.assertEqual(await self.get_by_data_filter(int_val__gt=100), self.obj)
+        self.assertEqual(await self.get_by_data_filter(int_val__gte=100), self.obj)
+        self.assertEqual(await self.get_by_data_filter(int_val__lt=200), self.obj)
+        self.assertEqual(await self.get_by_data_filter(int_val__lte=200), self.obj)
+        self.assertEqual(await self.get_by_data_filter(int_val__range=[100, 200]), self.obj)
+
+        with self.assertRaises(DoesNotExist):
+            await self.get_by_data_filter(int_val__gt=1000)
+
+    async def test_json_float_comparisons(self):
+        self.assertEqual(await self.get_by_data_filter(float_val__gt=100.0), self.obj)
+        self.assertEqual(await self.get_by_data_filter(float_val__gte=100.0), self.obj)
+        self.assertEqual(await self.get_by_data_filter(float_val__lt=200.0), self.obj)
+        self.assertEqual(await self.get_by_data_filter(float_val__lte=200.0), self.obj)
+        self.assertEqual(await self.get_by_data_filter(float_val__range=[100.0, 200.0]), self.obj)
+
+        with self.assertRaises(DoesNotExist):
+            await self.get_by_data_filter(int_val__gt=1000.0)
+
+    async def test_json_string_comparisons(self):
+        self.assertEqual(await self.get_by_data_filter(val__contains="ord"), self.obj)
+        self.assertEqual(await self.get_by_data_filter(val__icontains="OrD"), self.obj)
+        self.assertEqual(await self.get_by_data_filter(val__startswith="wor"), self.obj)
+        self.assertEqual(await self.get_by_data_filter(val__istartswith="wOr"), self.obj)
+        self.assertEqual(await self.get_by_data_filter(val__endswith="rd1"), self.obj)
+        self.assertEqual(await self.get_by_data_filter(val__iendswith="Rd1"), self.obj)
+        self.assertEqual(await self.get_by_data_filter(val__iexact="wOrD1"), self.obj)
+
+        with self.assertRaises(DoesNotExist):
+            await self.get_by_data_filter(val__contains="doesnotexist")
+
+    async def test_date_comparisons(self):
+        self.assertEqual(
+            await self.get_by_data_filter(date_val=datetime(1970, 1, 1, 12, 36, 59, 123456)),
+            self.obj,
+        )
+        self.assertEqual(await self.get_by_data_filter(date_val__year=1970), self.obj)
+        self.assertEqual(await self.get_by_data_filter(date_val__month=1), self.obj)
+        self.assertEqual(await self.get_by_data_filter(date_val__day=1), self.obj)
+        self.assertEqual(await self.get_by_data_filter(date_val__hour=12), self.obj)
+        self.assertEqual(await self.get_by_data_filter(date_val__minute=36), self.obj)
+        self.assertEqual(
+            await self.get_by_data_filter(date_val__second=Decimal("59.123456")), self.obj
+        )
+        self.assertEqual(await self.get_by_data_filter(date_val__microsecond=59123456), self.obj)
+
+    async def test_json_list(self):
+        self.assertEqual(await self.get_by_data_filter(int_list__0__gt=0), self.obj)
+        self.assertEqual(await self.get_by_data_filter(int_list__0__lt=2), self.obj)
+
+        with self.assertRaises(DoesNotExist):
+            await self.get_by_data_filter(int_list__0__range=(20, 30))
+
+    async def test_nested(self):
+        self.assertEqual(await self.get_by_data_filter(nested__val="word2"), self.obj)
+        self.assertEqual(await self.get_by_data_filter(nested__int_val=456), self.obj)
+        self.assertEqual(
+            await self.get_by_data_filter(
+                nested__date_val=datetime(1970, 1, 1, 12, 36, 59, 123456)
+            ),
+            self.obj,
+        )
+        self.assertEqual(await self.get_by_data_filter(nested__val__icontains="orD"), self.obj)
+        self.assertEqual(await self.get_by_data_filter(nested__int_val__gte=400), self.obj)
+        self.assertEqual(await self.get_by_data_filter(nested__date_val__year=1970), self.obj)
+        self.assertEqual(await self.get_by_data_filter(nested__date_val__month=1), self.obj)
+        self.assertEqual(await self.get_by_data_filter(nested__date_val__day=1), self.obj)
+        self.assertEqual(await self.get_by_data_filter(nested__date_val__hour=12), self.obj)
+        self.assertEqual(await self.get_by_data_filter(nested__date_val__minute=36), self.obj)
+        self.assertEqual(
+            await self.get_by_data_filter(nested__date_val__second=Decimal("59.123456")), self.obj
+        )
+        self.assertEqual(
+            await self.get_by_data_filter(nested__date_val__microsecond=59123456), self.obj
+        )
+        self.assertEqual(await self.get_by_data_filter(nested__val__iexact="wOrD2"), self.obj)
+        self.assertEqual(await self.get_by_data_filter(nested__int_val__lt=500), self.obj)
+        self.assertEqual(await self.get_by_data_filter(nested__date_val__year=1970), self.obj)
+        self.assertEqual(await self.get_by_data_filter(nested__date_val__month=1), self.obj)
+        self.assertEqual(await self.get_by_data_filter(nested__date_val__day=1), self.obj)
+        self.assertEqual(await self.get_by_data_filter(nested__date_val__hour=12), self.obj)
+        self.assertEqual(await self.get_by_data_filter(nested__date_val__minute=36), self.obj)
+        self.assertEqual(
+            await self.get_by_data_filter(nested__date_val__second=Decimal("59.123456")), self.obj
+        )
+        self.assertEqual(
+            await self.get_by_data_filter(nested__date_val__microsecond=59123456), self.obj
+        )
+        self.assertEqual(await self.get_by_data_filter(nested__val__iexact="wOrD2"), self.obj)
+
+    async def test_nested_nested(self):
+        self.assertEqual(await self.get_by_data_filter(nested__nested__val="word3"), self.obj)