Streaming, Generators, & JSONLines

[rtbs-dev]

Wondering what you thought the most straight-forward way of serializing many pyserde objects, possibly coming from a generator, and then deserializing with e.g. another generator?

I typically use jsonl files, which are just newline-delimited json objects, for such cases. I noticed you use orjson for parsing, but also noticed the capability for streaming to/from jsonld isn't a part of the core orjson library, but rather a part of a wrapper library orjsonl. See orjson issue

Is the intended way to go here that a custom serializer would get made to pass to to_json and from_json? Or maybe it makes sense for PySerde to include orjsonl as (possible) default behavior when e.g. the top-level class is a typing.Generator? There's a lot of really nice features in that package, but I'm not sure what would be possible/intended behavior for pyserde there.

The other option is for me to manually stream json from serde, but I definitely would be missing a lot of functionality that way, and adding a lot of test-case surface to my codebase 😅

Thanks!

[yukinarit]

If it's useful for many people, I am okay to have it included in pyserde!