Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mashumaro incompatibility: node.external.partitions returned as string instead of dict #11044

Open
2 tasks done
thomasgjerdekog opened this issue Nov 25, 2024 · 6 comments · May be fixed by #11051
Open
2 tasks done

mashumaro incompatibility: node.external.partitions returned as string instead of dict #11044

thomasgjerdekog opened this issue Nov 25, 2024 · 6 comments · May be fixed by #11051
Labels
bug Something isn't working

Comments

@thomasgjerdekog
Copy link

Is this a regression in a recent version of dbt-core?

  • I believe this is a regression in dbt-core functionality
  • I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

In dbt 1.8.9 it seems like the behavior of node.external.partitions has changed. It used to return a list of dicts, but now it seems to return a list of json strings.
This seems to be causing problems for dbt-external-tables when trying to create external tables with partitions defined.

I'm using this macro to compare the behaviors:

{% macro get_partitions() -%}
    {% set source_nodes = graph.sources.values() if graph.sources else [] %}
    {% for node in source_nodes %}
        {% if node.external %}
            {% if node.external.partitions %}
                {{print(node.external.partitions)}}
            {% endif %}
        {% endif %}
    {% endfor %}
{%- endmacro %}

dbt 1.8.8:

dbt run-operation get_partitions
11:23:26  Running with dbt=1.8.8
11:23:27  Registered adapter: snowflake=1.8.4
11:23:31  Found 1412 models, 2 snapshots, 6 seeds, 5 operations, 129 data tests, 619 sources, 2 exposures, 730 macros
[{'name': 'dl_partition', 'data_type': 'string', 'expression': "<redacted>"}]

dbt 1.8.9:

dbt run-operation get_partitions
11:21:38  Running with dbt=1.8.9
11:21:38  Registered adapter: snowflake=1.8.4
11:21:43  Found 1396 models, 2 snapshots, 6 seeds, 5 operations, 129 data tests, 611 sources, 2 exposures, 735 macros
['{\'name\': \'dl_partition\', \'data_type\': \'string\', \'expression\': "<redacted>"}']

Expected/Previous Behavior

I expect node.external.partitions to be a list of dicts so that the values can be accessed by looping the list and addressing with i.e. partition.name

Steps To Reproduce

  1. Install dbt 1.8.9
  2. Create a yaml source file called testtable.yml with external and partition set. I.e.:
version: 2
sources:
- name: TEST
  schema: STAGE
  tables:
  - name: TABLE
    external:
      partitions:
      - name: dl_partition
        data_type: string
        expression: split_part(METADATA$FILENAME, '/', 2)
  1. Create a test macro called get_partitions.sql:
{% macro get_partitions() -%}
    {% set source_nodes = graph.sources.values() if graph.sources else [] %}
    {% for node in source_nodes %}
        {% if node.external %}
            {% if node.external.partitions %}
                {{print(node.external.partitions)}}
            {% endif %}
        {% endif %}
    {% endfor %}
{%- endmacro %}
  1. Run the macro:
    dbt run-operation get_partitions

Relevant log output

No response

Environment

- OS: Windows 11 23H2/Ubuntu 22.04.5 LTS
- Python: Python 3.12.4
- dbt (working version): 1.8.8
- dbt (regression version): 1.8.9

Which database adapter are you using with dbt?

snowflake

Additional Context

No response
@thomasgjerdekog thomasgjerdekog added bug Something isn't working regression triage labels Nov 25, 2024
@dbeatty10
Copy link
Contributor

dbeatty10 commented Nov 26, 2024 •
edited
Loading

Thanks for reporting this @thomasgjerdekog !

I'm not seeing an obvious code difference between dbt-core 1.8.8 and 1.8.9 that explains what you're seeing, so I'm guessing it is via some dependency.

Could you share all the dbt versions for your dependencies in each of those different environments?

Assuming that you're using pip to install dbt-core, you can run something like this to get those versions:

pip list | grep '^dbt-'

Output might look something like this:

dbt-adapters               1.4.0
dbt-common                 1.7.0
dbt-core                   1.8.8
dbt-postgres               1.8.2
dbt-semantic-interfaces    0.5.1

@dbeatty10 dbeatty10 removed the triage label Nov 26, 2024
@thomasgjerdekog
Copy link
Author

Hi @dbeatty10
Sure!
Old (working):

pip freeze | grep "^dbt-"
dbt-adapters==1.7.0
dbt-common==1.10.0
dbt-core==1.8.8
dbt-extractor==0.5.1
dbt-osmosis==0.12.8
dbt-semantic-interfaces==0.5.1
dbt-snowflake==1.8.4

New (not working):

pip freeze | grep "^dbt-"
dbt-adapters==1.10.2
dbt-common==1.13.0
dbt-core==1.8.9
dbt-extractor==0.5.1
dbt-osmosis==0.12.8
dbt-semantic-interfaces==0.5.1
dbt-snowflake==1.8.4

I tried experimenting a bit with downgrading the dbt-adapters and dbt-common packages prior to submitting this issue, but I wasn't able to resolve the install conflicts since dbt-core 1.8.9 depends on protobuf >= 5.0 while versions of dbt-adapters prior to 1.10.0 depend on protobuf < 5.0

@taylordouglas
Copy link

taylordouglas commented Nov 26, 2024 •
edited
Loading

Running into the same issue on dbt-core 1.7.18 and dbt-redshift 1.7.7 with dbt-external-tables 0.10.0.

It looks like pinning mashumaro to 3.13.1 is fixing the issue for us.

@insectkorea
Copy link

Pinning mashumaro also worked for us.

@thomasgjerdekog
Copy link
Author

thomasgjerdekog commented Nov 27, 2024 •
edited
Loading

Edit: Sorry, seems like I had to clear my venv.

Pinning mashumaro to 3.13.1 worked for me as well.

PS: 3.14 also works

@gshank gshank changed the title [Regression] node.external.partitions returned as string instead of dict mashumaro incompatibility: node.external.partitions returned as string instead of dict Dec 2, 2024
@gshank gshank removed the regression label Dec 2, 2024
@gshank gshank linked a pull request Dec 3, 2024 that will close this issue
Open
5 tasks
@kendalldyke14 kendalldyke14 mentioned this issue Dec 13, 2024
Closed
3 tasks
@dataders
Copy link
Contributor

flagging that multiple users of dbt-external-tables are experiencing this and reported in dbt-labs/dbt-external-tables#325

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix code to work with mashumaro 3.15 dbt-labs/dbt-core
6 participants
@gshank @dataders @insectkorea @dbeatty10 @taylordouglas @thomasgjerdekog