Sign PyPI releases

[maltfield]

Describe the bug

When a user downloads this python module using pip, there is no cryptographic authenticity or integrity validation to protect the user from a MITM attack.

Therefore, this project is making any other projects that obtain the b2 module via pip in their build process vulnerable to a watering hole attack.

Expected behavior

A developer should have a mechanism to cryptographically verify the integrity and authenticity of b2 when obtaining it through pip.

To Reproduce

pip install b2

Additional context

Possible solutions include:

1. Using the --sign argument of twine when uploading packages to PyPI

2. Publishing a cryptographically signed document (ideally using gpg) listing the hashes for all packages uploaded to PyPI, which users can then pass into pip using the --hash argument

[maltfield]

Test showing that the release is not signed

user@disp1874:~/Downloads$ wget https://files.pythonhosted.org/packages/e7/8a/db65b2aee78993e3e60d1b149a7fe5fbf83a729ea7b28292b53e0e42943d/b2-3.0.1.tar.gz
--2021-08-26 20:01:40--  https://files.pythonhosted.org/packages/e7/8a/db65b2aee78993e3e60d1b149a7fe5fbf83a729ea7b28292b53e0e42943d/b2-3.0.1.tar.gz
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.37.63, 2a04:4e42:9::319
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.37.63|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 77626 (76K) [application/x-tar]
Saving to: ‘b2-3.0.1.tar.gz’

b2-3.0.1.tar.gz     100%[===================>]  75.81K  --.-KB/s    in 0.04s   

2021-08-26 20:01:40 (1.84 MB/s) - ‘b2-3.0.1.tar.gz’ saved [77626/77626]

user@disp1874:~/Downloads$ wget https://files.pythonhosted.org/packages/e7/8a/db65b2aee78993e3e60d1b149a7fe5fbf83a729ea7b28292b53e0e42943d/b2-3.0.1.tar.gz.asc
--2021-08-26 20:02:07--  https://files.pythonhosted.org/packages/e7/8a/db65b2aee78993e3e60d1b149a7fe5fbf83a729ea7b28292b53e0e42943d/b2-3.0.1.tar.gz.asc
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.37.63, 2a04:4e42:9::319
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.37.63|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-08-26 20:02:07 ERROR 404: Not Found.

user@disp1874:~/Downloads$ 

For more info, see:

• https://stackoverflow.com/questions/62274214/how-do-i-validate-a-pip-packages-pgp-key/62687869#62687869

[maltfield]

For best-practices on creating/maintaining gpg keys and release signing in-general, please see:

1. https://infra.apache.org/release-signing
2. https://docs.opendev.org/opendev/system-config/latest/signing.html
3. https://wiki.debian.org/Subkeys
4. https://riseup.net/en/security/message-security/openpgp/best-practices

[maltfield]

For examples of other developers on GitHub that fixed this by signing their packages on PyPI using twine --sign, see:

• Sign PyPI releases vpelletier/python-libusb1#54
• Sign PyPI Releases thinkst/opencanary#122

[ppolewicz]

You should generally not use wget to download packages from pypi, but even if you do that, if you use https, there is no way that the package that you'll receive will be corrupted by MITM attack, unless pypi itself is hacked or the attacker is an alien using a quantum computer to break through TLS. We also use a secure protocol when uploading the release.

This doesn't mean we won't start signing pypi releases, I think we will just because it's a nice thing to do, my point is just that there is no actual MITM vulnerability here unless the user follows an unsupported installation flow AND uses a dangerous option such as wget --no-check-certificate. Or if pypi itself is hacked.

It seems that you are familiar with the subject and also you seem to care about those signatures. Would you create a PR adding the signature creation and upload to our continous deployment? It's pretty standard, based on Github Actions, you could maybe even find a reusable component that does the trick without writing a single line of code.

[maltfield]

if you use https, there is no way that the package that you'll receive will be corrupted by MITM attack...

Sorry, but https doesn't always protect you from MITM attacks (and, no, it doesn't require a quantum computer or aliens). The problem isn't the crypto; it's the PKI. There's plenty of historic examples of CAs in the root stores loosing control of their root certs. For more info, see:

• https://security.stackexchange.com/questions/234052/where-can-i-find-a-list-of-all-government-agencies-with-cas-in-pki-root-stores

In the past, the maintainers of these root stores have removed CA's root certs for corporations who had lost control of their root certificates. [8][9] Similarly, there have been several historic controversies with CAs operated by government agencies. [10][11] And also with private entities that have been accused of being malicious actors (cyber-mercenaries) on behalf of governments. [12]
[8]: https://en.wikipedia.org/wiki/DigiNotar#Issuance_of_fraudulent_certificates
[9]: https://en.wikipedia.org/wiki/NortonLifeLock#Sustaining_digital_certificate_security
[10]: https://bugzilla.mozilla.org/show_bug.cgi?id=511380
[11]: https://bugzilla.mozilla.org/show_bug.cgi?id=1567114
[12]: https://www.eff.org/deeplinks/2019/02/cyber-mercenary-groups-shouldnt-be-trusted-your-browser-or-anywhere-else

Note also that HSTS is TOFU and therefore is useless unless there's browser persistence (which doesn't apply to pip anyway). And HPKP has been deprecated (sadly).

Would you create a PR adding the signature creation and upload to our continous deployment?

CI/CD is great, but you should not be keeping your private keys hot on a server. Major releases should be signed manually by a key stored somewhere safer (ideally cold storage).

Personally, I have two distinct keys: a "pre-release key" that I use to sign builds from my CI pipeline and a "release key" that I use to sign my actual releases after I merge into the release branch.

[ppolewicz]

I remember https://www.imperialviolet.org/2011/05/04/pinning.html being a non-TOFU solution, is github not covered by this?

I don't quite understand why release signing should not be automated. Like three people have access to those secrets (+we trust github). Is trusting github the problem you would like to address?

[maltfield]

Is trusting github the problem you would like to address?

You shouldn't trust GitHub, and it's not necessarily a problem with Microsoft. You just shouldn't trust any infrastructure in-between the endpoints. That's one reason why you sign releases (and use reproducible builds) -- it eliminates an enormous vector of risk.

• https://www.qubes-os.org/faq/#should-i-trust-this-website

Chrome's cert pinning only works in Chrome -- and afaik for only Google-owned properties. It's a Google thing they started doing since they happen to be a huge website and also their own web browser.

[ppolewicz]

Ok so if I got this right, the idea here is that you would like us to protect our users from:

1. Someone with a compromised CA (perhaps a hostile government) injecting a malicious binary via MITM when downloading package from pip/binary release to trick the user to run it
2. Github (hostile/hacked/coerced by govt. employee or just hacked infrastructure) signing a malicious binary and releasing it (perhaps only for some users, so that nobody finds out) so that a target user is tricked to run it

and the solution to 1. would be to sign the binary and pip releases, while the solution to 2. would be for the maintainers fo the project to perform the signing manually. Did I get it right?

I think we could do 1), but I'm not sure about 2).

That second part is tedious to manage - we would have to kill CD and either have a very limited number of people with the private key or we would need the private key to be in many different places. Is there a system we could use that would allow every developer to have his key and use that to sign things, to avoid sharing the secret and also avoid SPOF, but which would allow for easy verification to see whether a developer authorized at the time has created the given release?

[maltfield]

Honestly the easiest & most common solution is to assign only one single person to own the release signing private keys in cold storage and use it to sign your releases on a release cycle. This person (and their key) can be rotated in the future as-needed (should be done infrequently and accompanied by a public Key Transition Statement that's signed by the old key and the new key). Yes, this introduces a SPOF.

To deploy this at-scale and reduce a SPOF, you can mint the release signing key on an HSM and program it to sign your releases only when M-of-N of your developers sign it with their own private keys.

• https://github.com/coreos/fero

[ppolewicz]

I am apalled by the supply chain attacks and I agree we should implement #744 (also in b2-sdk-python and logfury) as well as #745 (I'd think it'll happen in January, unless of course someone contributes it first). #746 is already implemented, we sign and notarize macos releases too.

I'd also support you in convincing maintainers of our dependencies, even help them a little bit, if they don't know how to set it up or if they are not convinced it's worthwhile.

The CD-less process you have described in last post is quite heavy and also only as strong as the review thoroughness of all of the signers is. At this very moment I am not ready to pay the (organizational) cost associated with that level of security. When circumstances change one day (the need for it increases or the cost of it decreases or the cost I can afford to pay increases) then this can be looked at again, but today I think the basic signatures in CD are enough.

Please observe that I'm an opensource contributor to the repository, not a Backblaze employee. If you convince Backblaze their opensource clients and libraries should go beyond basic supply chain attack protection and into the heavier version, I'm sure it'll happen. Would Backblaze actually benefit from the heavier version though? Will data-heavy users choose Backblaze B2 over a different provider for the increased level of security? If we can prove they will, that will make the decision very easy.

As far as I can see, AWS S3 client does not seem to use releases at all, just using simple tags instead while their current pypi release is not signed either.