Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor dbt utility for modularity #1095

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

austinweisgrau
Copy link
Collaborator

@austinweisgrau austinweisgrau commented Jul 15, 2024

These changes uncouple the code for the dbt utility that runs a dbt command from the code that logs dbt commands. This allows for modular use of different loggers and a very flexible ability to customize logging. This addresses the main concerns from the original PR (#841).

Breaking changes

This PR does cause breaking changes with the previous implementation of the dbt utility, but those changes are well worth it for a much cleaner and clearer interface.

Using the new implementation looks like:

import os
from pathlib import Path
from parsons.utilities.dbt import run_dbt_commands, dbtLoggerSlack, dbtLoggerPython

results = run_dbt_commands(
    commands=["dbt run", "dbt test"],
    dbt_project_directory=Path("/path/to/dbt/project"),
    loggers=[dbtLoggerPython, dbtLoggerSlack(webhook=os.environ["SLACK_WEBHOOK"])],
)

The prior implementation was potentially only compatible with Redshift, depending on how the user's dbt profiles.yaml was set up. The new implementation allows the dbt process to inherit the shell environment from the python parent process, and is therefore compatible with running dbt on any database, and the user can configure credential passing using environment variables (recommended) or any other method.

Improved documentation

There is now much more thorough documentation throughout the 4 modules containing the dbt utility code.

TO DO:

  • Touch up the dbtLoggerMarkdown text formatting a bit
  • dbtLoggerDatabase splits artifacts into a run table and a node table

@austinweisgrau austinweisgrau force-pushed the dbt_bq branch 6 times, most recently from 5faf4b2 to c066e86 Compare July 17, 2024 20:13
@austinweisgrau austinweisgrau force-pushed the dbt_bq branch 3 times, most recently from 72dbeef to 24565bd Compare September 21, 2024 01:31
@austinweisgrau austinweisgrau force-pushed the dbt_bq branch 3 times, most recently from 636e109 to d35ae67 Compare October 16, 2024 00:59
@austinweisgrau austinweisgrau changed the title Draft - refactor dbt utility for modularity Refactor dbt utility for modularity Nov 4, 2024
@elyse-weiss
Copy link
Contributor

I do not feel qualified to approve, but having the requirements.txt file updated as it is here, would be helpful for me separately!

@austinweisgrau
Copy link
Collaborator Author

The build step is failing due to an unrelated dependency conflict, I think it's only triggering on this PR and not others because this one updates the docs folder

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants