Avoid conflicts with multiple test pypi releases that have the same version

[marvinmarnold]

Currently, only the first build after a version bump will pass .github/workflows/publis-to-pypi-test.yml. Example: https://github.com/NebraLtd/hm-pyhelper/pull/14/checks?check_run_id=3694402894. One solution the docs mention is to use setuptools_scm but the implications of the switch need to be better understood.

[vpetersson]

Personally, i think some kind of overwrite option would be desirable for test PyPi to avoid countless bumps.

[shawaj]

@vpetersson don't think you can overwrite on pypi it's a limitation on their side.

But you can just use alpha/beta/release candidate releases no?

0.1
0.2a1
0.2a2
0.2b3
0.2rc4
0.2

Etc etc

[shawaj]

Having said that, this answer is interesting https://stackoverflow.com/a/63944201/1965921

But TBH, I don't see how that's any easier than just incrementing an alpha / beta release.

Or incrementing a micro number instead like 0.2.1, 0.2.2 etc etc

Could use some automated versioning tool to do this without manual intervention also

Ref https://www.python.org/dev/peps/pep-0440/

[vpetersson]

The reason why i don't like manually bumping is that it takes too many steps (tag, push tag, create release). When you just want to push a quick fix, that's too much effort.

Maybe a quick-and-dirty solution is to simply run sed on setup.py and append the git hash to the release.

[shawaj]

@vpetersson for test pypi you don't have to tag anything. You just update the version in setup.py

[marvinmarnold]

FWIW my issues with the manual bump are less to do with the manual intervention required, but more-so:

• Determining the correct time to bump. If you bump before code review and changes are requested, you'll have to bump the version again for any subsequent changes. The conflict I linked to in the OP was due to flake8 tests that I missed locally and only noticed on the Github workflow.
• More significantly, this creates path dependency between PRs. Version must be bumped based on merge order, which is not known at the time a PR is opened.

[shawaj]

@marvinmarnold maybe it's best to only build testpypi when tagging rather than automatically?

[marvinmarnold]

Yea, that's prob the easiest thing to do now. @vpetersson I'm going to pull this into the current sprint because its small work but a big pain in the butt.

[shawaj]

or something similar to NebraLtd/hm-pktfwd#61

[Kerrryu]

Currently looking into a way that we can avoid the conflicts and have a way to automatically bump the version number. I found this https://github.com/c4urself/bump2version/#installation which seems to handle all of it for you in one command. It will bump the version with options for major/minor/revision and handles the git tags as well!

I'll try to look into it on a staging environment to see how well it works before I make the decision to implement it on prod

[Kerrryu]

I've created a pull request with the bump2version fix
#48

[marvinmarnold]

Closed by #48

[shawaj]

IMO this isn't really fixed. The bump2version, whilst it sort of does what we want, it's very limited and feels to me more like a stop gap.

Also, if we are going to be using this even for a short period of time, the auto version bumping should only be for patch level not minor.

Otherwise we are going to be bumping minor versions even for things as basic as a typo fix. Which should at best be a patch.

I feel like to be properly fixed this needs something more robust that takes into account the actual flow of PyPI (alpha, beta, RC releases) and that the major, minor, patch is automatically generated not arbitrarily but by actual human intervention/tagging...

For example:
Change-type: patch

In a commit message. Or something to that effect.

[Kerrryu]

@shawaj I can take a look and see if I can do something with a commit but I will change this to update it by patch version until then, I know bump2version supports the different channels of PyPI such as alpha, beta, and release so I can also include that in the commit message as well. I created a PR for having bump2version only run on the master branch here #51

[vpetersson]

IMO this isn't really fixed. The bump2version, whilst it sort of does what we want, it's very limited and feels to me more like a stop gap.

Also, if we are going to be using this even for a short period of time, the auto version bumping should only be for patch level not minor.

Otherwise we are going to be bumping minor versions even for things as basic as a typo fix. Which should at best be a patch.

I feel like to be properly fixed this needs something more robust that takes into account the actual flow of PyPI (alpha, beta, RC releases) and that the major, minor, patch is automatically generated not arbitrarily but by actual human intervention/tagging...

For example: Change-type: patch

In a commit message. Or something to that effect.

From a purist perspective if this was a widely used library, I would agree. That is not the case here, as we are just using it internally. We need to be pragmatic. The goal right now is to increase velocity and be able to iterate fast. In six to twelve months when things are stable and we have a more regular release cadence with release notes etc, but this is very much premature optimization and will just add a lot more complexity.

[marvinmarnold]

@SebastianMaj it looks like there was a regression. We've reintroduced test pypi conflicts, see: https://github.com/NebraLtd/hm-pyhelper/runs/4143321632?check_suite_focus=true

[Kerrryu]

I've got the workflow working in a local environment: https://github.com/SebastianMaj/hm-pytester

The current issue we are facing now is master branch is a protected branch and it's preventing any pushes to it unless a review is done beforehand. This is preventing bump2version from pushing the change to setup.py (bumping the version number in the file) in the workflow, I am looking into possible solutions to unblock this:

Solution 1 (Easiest implementation but least secure):
We can use a git token with admin privileges to push the change via workflow to ignore the protected branch rule, this would be the easiest to implement but it may pose some security issues to allow the automated workflow to push to master directly without review. Although the code only bumps one file so we are aware of what is being changed it should be ok but it's still something to be aware of.

Solution 2:
Manual bumping of the version number in setup.py and then using semantic versioning via Github action to bump version. This would completely remove bump2version and rely on the dev to bump the version manually on setup.py and continuous deployment will handle the git tag and release.

Solution 3:
Since we can't push the changes due to branch being protected we can use semantic versioning via Github Actions and instead of pushing the change to setup.py we can just rely on the local files present on the runner handling the action task, the runner will check for the latest version tag and change the setup.py version locally to use that version then a build will be done and uploaded, this will mean developers don't have to interact with the file at all and the version number present in setup.py means nothing for the release build as it will all be handled.

Workflow for Solution 3:

• Developer pushes code change
• Developer creates PR and merges into master
• The workflow will then bump up the version using Github Actions via Semantic Versioning
• This will then trigger another action to pull this version then update the version in setup.py (This won't be pushed but just a local change on the runner handling the action)
• Then we will do a py wheels build and upload it
• Then do a new release based on this and upload the binaries that were just built
• This will then trigger the release workflow to upload to PyPi and we will add the same steps of ensuring the version is correct on that build

Favored Versioning Action:
Semantic Version Action

Alternative Versioning Action:
Github Tag Action

Both of these are great and allow control of the version release via commit message (Ex: major, minor, patch) which is exactly what we are looking for.

Potential Blockers:
Will github allow us to push the tag via action or will this also be affected by the protected branch. I believe it should be fine as we've bumped the tag version before so this should not be a blocker for this issue.

[shawaj]

copying in from #78

@marvinmarnold FYI re your comment in #77 it actually works fairly well now, as long as you have updated the version in setup.py before merging to master from a PR, it will then upload to TestPyPI after that merge, and then to PyPI mainline when you publish the release.

So perhaps not fully fixing #20 but not particularly cumbersome either