Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(java/python): new xlang type system spec implementation #1690

Open
wants to merge 57 commits into
base: main
Choose a base branch
from

Conversation

chaokunyang
Copy link
Collaborator

@chaokunyang chaokunyang commented Jun 16, 2024

What does this PR do?

This PR implements a new type system for xlang serialization between java and python.

The changes includes:

  • Refine type system spec: added new types:
    • POLYMORPHIC_STRUCT = 16,
    • COMPATIBLE_STRUCT = 17,
    • POLYMORPHIC_COMPATIBLE_STRUCT = 18,
    • NAMED_STRUCT = 19,
    • NAMED_POLYMORPHIC_STRUCT = 20,
    • NAMED_COMPATIBLE_STRUCT = 21,
    • NAMED_POLYMORPHIC_COMPATIBLE_STRUCT = 22,
    • EXT = 23,
    • POLYMORPHIC_EXT = 24,
    • NAMED_EXT = 25,
    • NAMED_POLYMORPHIC_EXT = 26
  • Added a new XtypeResolver in java to resolve xlang types
  • Support pass type into to resolve type ambiguation such as ArrayList/Object[] in java. Users can serialize(List.of(1, 2, ,3)) and deserialize it into array by deserialize(bytes, Integer[].class)
  • Refactor pyfury serialization by moving type resolver into python code from cython, this will make debug more easy and reduce code duplciation, it also speed serialization performance.
  • golang xtype serialization test are disabled, it will be reenabled after new type system is implemented in golang

Related issues

Does this PR introduce any user-facing change?

  • Does this PR introduce any public API change?
  • Does this PR introduce any binary protocol compatibility change?

Benchmark

@chaokunyang chaokunyang marked this pull request as draft June 16, 2024 14:40
chaokunyang added a commit that referenced this pull request Jul 13, 2024
## What does this PR do?

This PR moves `org.apache.fury.reflect.Types` into TypeRef. Types is
used in Fury type system, by merge `org.apache.fury.reflect.Types`, we
can reduce ambiguation in fury type system

## Related issues
#1553 
#1690 

## Does this PR introduce any user-facing change?

<!--
If any user-facing interface changes, please [open an
issue](https://github.com/apache/fury/issues/new/choose) describing the
need to do so and update the document if necessary.
-->

- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?


## Benchmark

<!--
When the PR has an impact on performance (if you don't know whether the
PR will have an impact on performance, you can submit the PR first, and
if it will have impact on performance, the code reviewer will explain
it), be sure to attach a benchmark data here.
-->
@chaokunyang chaokunyang force-pushed the new_xlang_sepc_impl branch from 86a35c3 to a3d4d9d Compare July 13, 2024 11:41
@chaokunyang
Copy link
Collaborator Author

This pr is relatively large, can we briefly describe what this pr does?

I haven't finish it, this PR mainly reimplement xlang serialization with new protocol. Detailed description will be added in next days.

@chaokunyang chaokunyang changed the title feat: new xlang spec impl feat(java/python): new xlang spec impl Dec 20, 2024
@chaokunyang chaokunyang changed the title feat(java/python): new xlang spec impl feat(java/python): new xlang type system spec implementation Dec 23, 2024
@chaokunyang chaokunyang marked this pull request as ready for review December 23, 2024 04:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants