All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- Use of
structlog~=24.1.0for standard log outputs - Verified Schemachange against Python 3.12
- Support for connections.toml configurations
- Support for supplying the authenticator, private key path, token path, connections file path, and connection name via the YAML and command-line configurations.
- Refactored the main cli.py into multiple modules - config, session.
- Updated contributing guidelines and demo readme content to help contributors setup local snowflake account to run the github actions in their fork before pushing the PR to upstream repository.
- Removed tests against Python 3.8 End of Life on 2024-10-07
- Command-line vars are now merged into YAML vars instead of overwriting them entirely
- Improved unit test coverage
- Added Session ID as part of the initial connection successful message to be visible in the logs
- Aligning with snowflake identifier requirements in the configuration settings
- Fixed the bug with
Missing default warehouse - Fixed Demo examples resulting from change in public data set location
- Removed pandas library dependency to improve schemachange install footprint
- Updated Github Actions Workflow to check PRs and Merges does not break the demo examples.
- Updated Docs related to latest Demo content included in schemachange
- Updated pandas version dependency
- Pinned NumPy version dependency
- Allow passing snowflake schema as config or CLI parameter
- Fixed bug introduced in version 3.5.0 where the session state was not reset after a user script was run. This resulted in schemachange updates to the metadata table failing in some cases. schemachange will now reset the session back to the default settings after each user script is run
- Updated the pytest GitHub Actions workflow
- Cleaned up whitespace and formatting in files for consistency
- Updated README file
- Added new dependency review GitHub Actions workflow
- Added badges for pytest and PyPI
- Fixed authentication workflow to check for authenticator type first, then for Key pair and finally default to password authentication.
- Fixed the
Dockerfile-srcconfiguration to build a docker image from source code. - Updated README file.
- Updated version dependencies for
snowflake-connector-pythonandpyyaml.
- Fixed bug (from the 3.5.0 release) that caused a crash when using verbose logging.
- Fixed a bug when handling default values from the command line with arguments defined as
action='store_true'(create-change-history-table, auto-commit, verbose, and dry-run).
- Added support for Oauth and external browser and Okta authentication methods.
- Added
--oauth-configto accept values for oauth configuration.
- Inverted Program Call sequence and refactored all snowflake interactions into a Class. Class now persists connection accross all interactions and updates the snowflake query tag session variable as scripts are executed.
- Cleaned up argument passing and other repetitive code using dictionary and set comparisons for easy maintenance. (Converted variable names to a consistent snake_case from a mix of kebab-case and snake_case)
- Fixed change history table processing to allow mixed case names when '"' are used in the name.
- Moved most error, log and warning messages and query strings to global or class variables.
- Updated readme to cover new authentication methods
- Updated
snowflake-connector-pythondependency to version 2.8. This should address errors with result batching in thefetch_r_script_checksummethod when users have a lot of scripts in their project.
- Added a new optional parameter
--query-tagto append a string to the QUERY_TAG that is attached to every SQL statement executed
- Added filtering of secrets when vars are displayed on the console.
- Added filtering of secrets for deploy command when SQL statements are displayed as part of verbose output.
- Changed vars to be pretty printed to the console.
- Changed demo citibike_jinja to demonstrate secret filtering.
- Updated the Jinja templating engine section of the README.md to document Jinja autoescaping status and added warning about untrusted input.
- Updated the table of contents section of the README.md to included missing sections.
- Added
env_varJinja function support to migration templates. - Backed out Jinja autoescape change from 3.3.2. The default is now to have it disabled (using autoescape=False).
- Configured Jinja to escape inputs to templates (using autoescape=True). This helps protect rendered templates against XSS and other vulnerabilities
- Project is now configured with setup.cfg. There should be no change to package users.
- Added processing of schemachange-config.yml with jinja templating engine.
- Included new Jinja function env_var for accessing environmental variables from the config file.
- Restored CLI tests, hopefully less fragile now.
- Added GitHub CI workflow to run unit tests and a basic execution test.
schemachange.cli.mainis now defined asdef main(argv: List[str]=sys.argv), to allow consumers to pass a list of arguments easily.
- Jinja Template Engine was not recognising scripts in subfolders on Windows machines. Jinja was expecting the paths to follow a unix style ie SQL/V2.0.0__ADHOC_SCRIPT.sql but on Windows machines this was being passed through as SQL\V2.0.0__ADHOC_SCRIPT.sql.
- Removed fragile unit tests in test_main.py.
- Added support for jinja templates. Any file ending .sql or .sql.jinja will be processed using the Jinja engine
- Added a new optional parameter
--modules-folderto specify where common jinja template, macro or include files reside
- Added a new optional parameter
- Added new subcommands render and deploy
- The render command can be used to display the final script to the command line.
- The existing functionality moved to a new deploy subcommand
- Fall back behaviour to assume deploy sub command if none provided
- Added reserved variable name
schemachangeand an error will now be raised if supplied by the user via --vars
- Added check for duplicate filenames. An error will now be generated should two scripts in different folders have the same name. The old behaviour resulted in just the last found script being included for execution.
- Loosen dependency version constraints
- Fix crash on dry run with no change history table
- Added support for configuring schemachange through a
schemachange-config.ymlYAML file!- You can now invoke schemachange without supplying any command line arguments if you use the config file
- The filename for the config file is expected to be
schemachange-config.yml - Command line arguments override values in the config files
- This also makes it easier to pass variables to your scripts
- For more details please see the README for more details.
- Added a new optional parameter
--config-folderto specify where your config file resides
- Add Always script type (scripts that begin with the letter 'A'). Always scripts are executed with every run of schemachange
- Fix repeatable scripts to only execute if there is a change in the script. Repeatable scripts will not be executed with every run anymore!!!
- IMPORTANT: If you were relying on the existing behavior, please rename those scripts to start with the letter 'A' (see above)
- Updated versioned script filename parsing logic to use lazy regex matching for splitting version tags
- This addresses a bug with having double underscores (__) in the description
- Added support for unencrypted private keys
SNOWFLAKE_PRIVATE_KEY_PASSPHRASEenvironment variable is no longer required if the key is not encrypted
- Added 'schemachange' as an application parameter when connecting, to enable usage stats.
- Remove requirement for running user to have CREATE SCHEMA privileges for --create-change-history-table when schema already exists (#42)
- Bumped snowflake-connector-python to 2.4.2 and relaxed the pip dependency to ~= (pick the latest patch release 2.4.X on install)
- Bumped the docker container python version to 3.9 as snowflake-connector-python now supports this as of 2.4
- Created a 'Dockerfile-src' for users wanting to build schemachange themselves from source in Docker
- Renamed snowchange to schemachange
- Please be advised that the new default path for the change history table is now
METADATA.SCHEMACHANGE.CHANGE_HISTORY(the schema name has been changed fromSNOWCHANGEtoSCHEMACHANGE). The easiest thing to do is rename the metadata schema in Snowflake. The other option is to leave it as is and use the--change-history-table(or-c) parameter to override the default.
- Added a new optional parameter
--dry-runto preview scripts that will be applied by snowchange.
- Added a Snowflake query tag to all snowchange queries. If the query is issued by the tool the tag will be "snowchange #.#.#" (e.g. "snowchange 2.7.0") and if the query is part of a change script the tag will be "snowchange #.#.#;<script name>" (e.g. "snowchange 2.7.0;V1.1.1__initial_database_objects.sql")
- Added a new optional parameter
-dor--snowflake-databaseto specify the default database to use (which can always be overridden in an individual change script) - Added a new optional parameter
--create-change-history-tableto create the change history table if it does not exist
- The default mode of operation now is to not create the change history table if it doesn't exist. Instead, now if the history table doesn't exist the tool will fail by default.
- Added a check to see if the change history table exists before trying to create it, which is now also dependent upon the
--create-change-history-tablecommand line argument. - Cleaned up command line argument descriptions
- Removed the ability for snowchange to create the database for the change history table. snowchange now requires the database to be created ahead of time.
- Support for encrypted key pair authentication
- PyPI package support so that snowchange can be installed through pip
- Dockerfile for building Docker images
- Allow for .SQL file extensions (common in Windows environments) in addition to .sql extensions
- Support for "repeatable" scripts such as stored procedures, functions etc
- Support for variables in change scripts (following a Jinja expression syntax)! See the README for more details.
- A new optional parameter
--varswhich accepts a JSON formatted string of variables and values (e.g.{"variable1": "value1", "variable2": "value2"})
- Add the Snowflake account name to the script output to provide more log context
- Removed the deprecated
--snowflake-regionparameter. Instead, use the-aor--snowflake-accountaccount parameter. See Usage Notes for the account Parameter (for the connect Method) for more details on how to structure the account name.
- Support for cross-database dependencies!
- Support for override the location and name of the change history table with the new parameter
--change-history-table(or-c) - Change log for snowchange project (this CHANGELOG.md file)
- The required project folder structure, removing the database folder convention
- Where the change history table gets created and how it gets named
- The schema of the change history table (removed INSTALLED_RANK and renamed/reordered a few columns)
- Updated the getting started section of the README.md to make the getting started steps more clear
- The ability for snowchange to create user databases directly (now the user must explicity do so in their change scripts)
- The
--environment-nameand--append-environment-nameparameters