Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codegen dataclassess #79

Merged
merged 18 commits into from
Dec 6, 2023

Conversation

Daquiver1
Copy link
Contributor

No description provided.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 15, 2023
@Daquiver1 Daquiver1 force-pushed the codegen_dataclassess branch from 4070a21 to 2973c17 Compare November 24, 2023 12:15
projects/jdwp/tests/test_dataclass_generator.py Outdated Show resolved Hide resolved
projects/jdwp/tests/test_primitve_type.py Outdated Show resolved Hide resolved
projects/jdwp/codegen/dataclass_generator.py Outdated Show resolved Hide resolved
Copy link
Contributor

@michalgr michalgr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also need to emit dataclasses for the nested structs. We'll need some code to visit/iterate over all nested structs.

projects/jdwp/codegen/dataclass_generator.py Outdated Show resolved Hide resolved
projects/jdwp/codegen/dataclass_generator.py Outdated Show resolved Hide resolved
Copy link
Contributor

@michalgr michalgr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend separating struct name computation from python type computation. We need functions/classes doing the following:

  1. Walking Structs to retrieve nested Structs
  2. Assigning names to Structs
  3. Computing types of fields given names of all the Structs involved
  4. Generating code for data classes
    If possible it would be great to minimise dependencies between them.

For example I would start by writing a generator recursively walking structs, something like this:

StructLink = typing.Tuple[Struct, Field, Struct]
def nested_structs(root: Struct) -> typing.Generator[StructLink, None, None]:
    for field in root.fields:
        type = field.type
        match type:
            case Array():
                yield root, field, type.element_type
                yield from nested_structs(type.element_type)
            case TaggedUnion():
                for struct in type.cases.values():
                    yield root, field, struct
                    yield from nested_structs(struct)
            case Struct():
                yield root, field, type
                yield from nested_structs(type)

Such generator would then be a base for the algorithm assigning names to structs and algorithm generating all the dataclasses we want. For example:

def compute_struct_names(root: Struct, name: str) -> Mapping[Struct, str]:
    names = {
        root: name,
    }
    for parent, field, nested in nested_structs(root):
        type = field.type
        match field.type:
            case Struct():
                names[nested] = f"{names[parent]}{field.name.captialize()}"
            case Array():
                ...
            case TaggedUnion:
                ....
    return names


# dedicated class to make it easier to pass all the necessary state around
class StructGenerator:
    def __init__(self, root: Struct, name: str):
        self.__struct_to_name = compute_struct_names(root, name)

    def __get_python_type_for(self, struct: Struct, field: Field) -> str:
        type = field.type
        match type:
            case Struct():
                return self. __struct_to_name[type]

            case Array():
                return f"typing.List[{self.__struct_to_name[type.element_type]}]"

            case TaggedUnion():
                ...

            case _:
                return types.python_type_for(type)

    def __generate_dataclass(self, struct: Struct) -> str:
        name = self.__struct_to_name[struct]
        class_def = ...
        return dedent(class_def)

    def generate(self):
        return [
            self.__generate_dataclass(nested)
            for _, _, nested in reversed(nested_structs(self.__root))
        ] + [self.__generate_dataclass(self.__root)]

projects/jdwp/codegen/dataclass_generator.py Outdated Show resolved Hide resolved
Copy link
Contributor

@michalgr michalgr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests seem to be failing.

@Daquiver1 Daquiver1 force-pushed the codegen_dataclassess branch from b6716db to 60331c0 Compare December 6, 2023 09:38
@michalgr michalgr merged commit 2e88acd into facebookexperimental:main Dec 6, 2023
10 checks passed
wekesa360 pushed a commit to wekesa360/ExtendedAndroidTools that referenced this pull request Dec 6, 2023
[jdwp] make jdwp schema immutable and hashable

[jdwp] make struct serialization synchronous

Codegen dataclassess (facebookexperimental#79)

* Wrote test to validate primitive type mapping

* Added test for type alias definition

* Built dataclasses code generator

* Built test for dataclassess code generator

* Changed to unittest

* Updated struct dataclass function to support all types

* feat: built dataclass generator

* Added copyright

* Added nested struct support

* Added test for nested struct

* Feat: update functions and put it in a class.

* chore: update tests

* Casted types to pass pyre check

* Updated types

* chore: Refactor typing.cast code

* Added copyright

* refactor: update tests to reflect new schema changes

* fix: Fix pyre typing errors

[jdwp] do not emit array length and union tag fields

[jdwp] map union tags and array lengths to int

[jdwp] fix type for union fields

[jdwp] fix names of fields containing spaces

[jdwp] generate projects.jdwp.runtime.structs

[jdwp] add tests importing components of the runtime library

implement async abstract methods, add TCP conn

update files

update conn and streams

add buck deps
wekesa360 pushed a commit to wekesa360/ExtendedAndroidTools that referenced this pull request Dec 7, 2023
[jdwp] make jdwp schema immutable and hashable

[jdwp] make struct serialization synchronous

Codegen dataclassess (facebookexperimental#79)

* Wrote test to validate primitive type mapping

* Added test for type alias definition

* Built dataclasses code generator

* Built test for dataclassess code generator

* Changed to unittest

* Updated struct dataclass function to support all types

* feat: built dataclass generator

* Added copyright

* Added nested struct support

* Added test for nested struct

* Feat: update functions and put it in a class.

* chore: update tests

* Casted types to pass pyre check

* Updated types

* chore: Refactor typing.cast code

* Added copyright

* refactor: update tests to reflect new schema changes

* fix: Fix pyre typing errors

[jdwp] do not emit array length and union tag fields

[jdwp] map union tags and array lengths to int

[jdwp] fix type for union fields

[jdwp] fix names of fields containing spaces

[jdwp] generate projects.jdwp.runtime.structs

[jdwp] add tests importing components of the runtime library

implement async abstract methods, add TCP conn

update files

update conn and streams

add buck deps

add jvm connection and implement jvm streams
wekesa360 pushed a commit to wekesa360/ExtendedAndroidTools that referenced this pull request Dec 7, 2023
Codegen dataclassess (facebookexperimental#79)

* Wrote test to validate primitive type mapping

* Added test for type alias definition

* Built dataclasses code generator

* Built test for dataclassess code generator

* Changed to unittest

* Updated struct dataclass function to support all types

* feat: built dataclass generator

* Added copyright

* Added nested struct support

* Added test for nested struct

* Feat: update functions and put it in a class.

* chore: update tests

* Casted types to pass pyre check

* Updated types

* chore: Refactor typing.cast code

* Added copyright

* refactor: update tests to reflect new schema changes

* fix: Fix pyre typing errors

[jdwp] do not emit array length and union tag fields

[jdwp] map union tags and array lengths to int

[jdwp] fix type for union fields

[jdwp] fix names of fields containing spaces

[jdwp] generate projects.jdwp.runtime.structs

[jdwp] add tests importing components of the runtime library

implement async abstract methods, add TCP conn

update files

update conn and streams

add buck deps

add jvm connection and implement jvm streams

add a base class for jdwp structs

[jdwp] generate projects.jdwp.runtime.structs

[jdwp] add tests importing components of the runtime library

fix fail build
wekesa360 pushed a commit to wekesa360/ExtendedAndroidTools that referenced this pull request Dec 7, 2023
Codegen dataclassess (facebookexperimental#79)

* Wrote test to validate primitive type mapping

* Added test for type alias definition

* Built dataclasses code generator

* Built test for dataclassess code generator

* Changed to unittest

* Updated struct dataclass function to support all types

* feat: built dataclass generator

* Added copyright

* Added nested struct support

* Added test for nested struct

* Feat: update functions and put it in a class.

* chore: update tests

* Casted types to pass pyre check

* Updated types

* chore: Refactor typing.cast code

* Added copyright

* refactor: update tests to reflect new schema changes

* fix: Fix pyre typing errors

[jdwp] do not emit array length and union tag fields

[jdwp] map union tags and array lengths to int

[jdwp] fix type for union fields

[jdwp] fix names of fields containing spaces

[jdwp] generate projects.jdwp.runtime.structs

[jdwp] add tests importing components of the runtime library

implement async abstract methods, add TCP conn

update files

update conn and streams

add buck deps

add jvm connection and implement jvm streams

add a base class for jdwp structs

[jdwp] generate projects.jdwp.runtime.structs

[jdwp] add tests importing components of the runtime library

fix fail build

[jdwp] fix type checker fails
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants