Skip to content

Commit

Permalink
chore: bump version to test release and ensure todos are addressed in…
Browse files Browse the repository at this point in the history
… cli
  • Loading branch information
z3z1ma committed Jan 2, 2025
1 parent c1b7563 commit e05488f
Show file tree
Hide file tree
Showing 12 changed files with 31 additions and 97 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ jobs:
- name: Build package
run: |
uv build --wheel .
uvx hatch build
- name: Publish package on PyPI
if: steps.check-version.outputs.tag
Expand Down
27 changes: 4 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,7 @@
[![Downloads](https://static.pepy.tech/badge/dbt-osmosis)](https://pepy.tech/project/dbt-osmosis)
![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-green.svg)
![black](https://img.shields.io/badge/code%20style-black-000000.svg)
[![Streamlit App](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://z3z1ma-dbt-osmosis-srcdbt-osmosisapp-v2-i0ico9.streamlit.app/)

[![Streamlit App](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://dbt-osmosis-playground.streamlit.app/)

[![Scc Count Badge](https://sloc.xyz/github/z3z1ma/dbt-osmosis/)](https://github.com/z3z1ma/dbt-osmosis/)
[![Scc Count Badge](https://sloc.xyz/github/z3z1ma/dbt-osmosis/?category=cocomo)](https://github.com/z3z1ma/dbt-osmosis/)
Expand Down Expand Up @@ -38,24 +37,12 @@ Hello and welcome to the project! [dbt-osmosis](https://github.com/z3z1ma/dbt-os

> Automatically generate documentation based on upstream documented columns
2. A highly performant dbt server which integrates with tools such as dbt-power-user for VS Code to enable interactive querying + realtime compilation from your IDE

2a. `dbt-osmosis server serve --project-dir ... --profiles-dir ...`

> Spins up a WSGI server. Can be passed --register-project to automatically register your local project
3. Workbench for dbt Jinja SQL. This workbench is powered by streamlit and the badge at the top of the readme will take you to a demo on streamlit cloud with jaffle_shop loaded (requires extra `pip install "dbt-osmosis[workbench]"`).
2. Workbench for dbt Jinja SQL. This workbench is powered by streamlit and the badge at the top of the readme will take you to a demo on streamlit cloud with jaffle_shop loaded (requires extra `pip install "dbt-osmosis[workbench]"`).

3a. `dbt-osmosis workbench --project-dir ... --profiles-dir ...`

> Spins up a streamlit app. This workbench offers similar functionality to the osmosis server + power-user combo without a reliance on VS code. Realtime compilation, query execution, pandas profiling all via copying and pasting whatever you are working on into the workbenchat your leisure. Spin it up and down as needed.
4. Diffs for data model outputs to model outputs across git revisions (🚧 this is in development)

4a. `dbt-osmosis diff -m some_model --project-dir ... --profiles-dir ...`

> Run diffs on models dynamically. This pulls the state of the model before changes from your git history, injects it as a node to the dbt manifest, compiles the old and modified nodes, and diffs their query results optionally writing nodes to temp tables before running the diff query for warehouses with performance or query complexity limits (👀 bigquery)
____

## Pre-commit
Expand All @@ -65,7 +52,7 @@ You can use dbt-osmosis as a pre-commit hook. This will run the `dbt-osmosis yam
```yaml title=".pre-commit-config.yaml"
repos:
- repo: https://github.com/z3z1ma/dbt-osmosis
rev: v0.11.11 # verify the latest version
rev: v1.0.1 # verify the latest version
hooks:
- id: dbt-osmosis
files: ^models/
Expand All @@ -84,8 +71,7 @@ I also expect there is some untapped value in the workbench that is only pending
Demo the workbench 👇
[![Streamlit App](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://z3z1ma-dbt-osmosis-srcdbt-osmosisapp-v2-i0ico9.streamlit.app/)
[![Streamlit App](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://dbt-osmosis-playground.streamlit.app/)
```sh
# NOTE this requires the workbench extra as you can see
Expand All @@ -97,7 +83,6 @@ dbt-osmosis workbench

Press "r" to reload the workbench at any time.


✔️ dbt Editor with instant dbt compilation side-by-side or pivoted

✔️ Full control over model and workbench theme, light and dark mode
Expand All @@ -106,7 +91,6 @@ Press "r" to reload the workbench at any time.

✔️ Data Profiler (leverages pandas-profiling)


**Editor**

The editor is able to compile models with control+enter or dynamically as you type. Its speedy! You can choose any target defined in your profiles yml for compilation and execution.
Expand All @@ -117,7 +101,6 @@ You can pivot the editor for a fuller view while workbenching some dbt SQL.

![pivot](/screenshots/osmosis_editor_pivot.png?raw=true "dbt-osmosis Pivot Layout")


**Test Query**

Test dbt models as you work against whatever profile you have selected and inspect the results. This allows very fast iterative feedback loops not possible with VS Code alone.
Expand All @@ -130,14 +113,12 @@ Profile your datasets on the fly while you develop without switching context. Al

![profile-data](/screenshots/osmosis_profile.png?raw=true "dbt-osmosis Profile Data")


**Useful Links and RSS Feed**

Some useful links and RSS feeds at the bottom. 🤓

![profile-data](/screenshots/osmosis_links.png?raw=true "dbt-osmosis Profile Data")


___

![graph](https://repobeats.axiom.co/api/embed/df37714aa5780fc79871c60e6fc623f8f8e45c35.svg "Repobeats analytics image")
16 changes: 0 additions & 16 deletions docker/Dockerfile.dev

This file was deleted.

29 changes: 0 additions & 29 deletions docker/docker-compose.yml

This file was deleted.

4 changes: 0 additions & 4 deletions docker/shell

This file was deleted.

3 changes: 0 additions & 3 deletions docker/shutdown

This file was deleted.

8 changes: 0 additions & 8 deletions docker/startup

This file was deleted.

6 changes: 3 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
[build-system]
requires = ["hatchling"]
requires = ["hatchling==1.26.3"] # hatch-vcs
build-backend = "hatchling.build"

[project]
name = "dbt-osmosis"
version = "1.0.0"
version = "1.0.1"
description = "A dbt utility for managing YAML to make developing with dbt more delightful."
readme = "README.md"
license = { text = "Apache-2.0" }
Expand Down Expand Up @@ -38,7 +38,7 @@ sqlite = ["dbt-sqlite>=1.0.0,<2"]
postgres = ["dbt-postgres>=1.0.0,<2"]

workbench = [
"streamlit>=1.20.0,<1.36.0",
"streamlit>=1.20.0,<1.30.0",
"streamlit-ace~=0.1.1",
"ydata-profiling~=4.12.1",
"feedparser~=6.0.11",
Expand Down
8 changes: 4 additions & 4 deletions src/dbt_osmosis/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ def wrapper(*args: P.args, **kwargs: P.kwargs) -> T:
help="If specified, we will skip merging upstrean meta keys to the model columns.",
)
@click.option(
"--skip-add-data-types", # TODO: make sure this is implemented
"--skip-add-data-types",
is_flag=True,
help="If specified, we will skip adding data types to the models.",
)
Expand Down Expand Up @@ -188,7 +188,7 @@ def wrapper(*args: P.args, **kwargs: P.kwargs) -> T:
help="If specified, will add inheritance for the specified keys. IE policy_tags",
)
@click.option(
"--output-to-lower", # TODO: validate this is implemented
"--output-to-lower",
is_flag=True,
help="If specified, output yaml file columns and data types in lowercase if possible.",
)
Expand Down Expand Up @@ -352,7 +352,7 @@ def organize(
help="If specified, we will skip merging upstrean meta keys to the model columns.",
)
@click.option(
"--skip-add-data-types", # TODO: make sure this is implemented
"--skip-add-data-types",
is_flag=True,
help="If specified, we will skip adding data types to the models.",
)
Expand Down Expand Up @@ -388,7 +388,7 @@ def organize(
help="If specified, will add inheritance for the specified keys. IE policy_tags",
)
@click.option(
"--output-to-lower", # TODO: validate this is implemented
"--output-to-lower",
is_flag=True,
help="If specified, output yaml file columns and data types in lowercase if possible.",
)
Expand Down
19 changes: 16 additions & 3 deletions src/dbt_osmosis/core/osmosis.py
Original file line number Diff line number Diff line change
Expand Up @@ -810,12 +810,12 @@ def create_missing_source_yamls(context: YamlRefactorContext) -> None:
)

def _describe(rel: BaseRelation) -> dict[str, t.Any]:
return {
s = {
"name": rel.identifier,
"description": "",
"columns": [
{
"name": name,
"name": name.lower() if context.settings.output_to_lower else name,
"description": meta.comment or "",
"data_type": meta.type.lower()
if context.settings.output_to_lower
Expand All @@ -824,6 +824,10 @@ def _describe(rel: BaseRelation) -> dict[str, t.Any]:
for name, meta in get_columns(context, get_table_ref(rel)).items()
],
}
if context.settings.skip_add_data_types:
for col in t.cast(list[dict[str, t.Any]], s["columns"]):
_ = col.pop("data_type", None)
return s

tables = [
schema
Expand Down Expand Up @@ -1183,6 +1187,12 @@ def _sync_doc_section(
for k, v in cdict.items():
if k == "description" and not v:
merged.pop("description", None)
elif (
k == "data_type"
and merged.get("data_type") is None
and context.settings.skip_add_data_types
):
pass
else:
merged[k] = v

Expand All @@ -1197,6 +1207,9 @@ def _sync_doc_section(
if not merged[k]:
merged.pop(k)

if context.settings.output_to_lower:
merged["name"] = merged["name"].lower()

incoming_columns.append(merged)

doc_section["columns"] = incoming_columns
Expand Down Expand Up @@ -1578,7 +1591,7 @@ def inject_missing_columns(context: YamlRefactorContext, node: ResultNode | None
node.unique_id,
)
gen_col = {"name": incoming_name, "description": incoming_meta.comment or ""}
if dtype := incoming_meta.type:
if (dtype := incoming_meta.type) and not context.settings.skip_add_data_types:
gen_col["data_type"] = dtype.lower() if context.settings.output_to_lower else dtype
node.columns[incoming_name] = ColumnInfo.from_dict(gen_col)

Expand Down
2 changes: 1 addition & 1 deletion src/dbt_osmosis/workbench/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
dbt-osmosis[workbench,duckdb]==1.0.0
dbt-osmosis[workbench,duckdb]==1.0.1
setuptools>=70
4 changes: 2 additions & 2 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit e05488f

Please sign in to comment.