Puller not working with (custom) system certs

[abergmeier]

We have custom certs at my company. These are installed in /usr/local/share/ca-certificates and work fine with Browsers and most tools (on Ubuntu 16.04.3).
Sadly puller is the exception. It fails with:

File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/__main__.py", line 93, in <module>
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/__main__.py", line 80, in main
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/containerregistry/client/v2_2/docker_image_.py", line 324, in __enter__
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/containerregistry/client/v2_2/docker_http_.py", line 191, in __init__
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/containerregistry/client/v2_2/docker_http_.py", line 217, in _Ping
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/containerregistry/transport/transport_pool_.py", line 62, in request
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/httplib2/__init__.py", line 1659, in request
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/httplib2/__init__.py", line 1399, in _request
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/httplib2/__init__.py", line 1319, in _conn_request
File "/home/user/.cache/bazel/_bazel_user/39b2d6f0d213152194c5e70c47d440c6/external/puller/file/puller.par/httplib2/__init__.py", line 1092, in connect
httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

Is there any way of telling that to use the system certificates?
If not, where does puller/httplib2 store the certificates?

[abergmeier]

Trying to fix this in google/containerregistry#50.

[calder]

This fixed the error for me:

sudo pip install httplib2.system-ca-certs-locater

[roblg]

I'm still investigating and trying to wrap my head around everything, but I believe we're experiencing issues (CERTIFICATE_VERIFY_FAILED) connecting to private registry secured by a certificate signed by a real root CA (Comodo) as well, so this issue may not be limited to custom certs. My current theory is that the root CA that signed our cert isn't bundled with httplib2, way down in the depths of containerregistry.

curl, docker push, my browser, and other non-containerregistry interactions with the private registry succeed w/ no issue.

Will report back with more info as I find it.

Update: Ultimately, I think our issue is the same as described above, even though it's not a custom cert -- we have a cert signed by a root CA that isn't included in the cacerts.txt bundled with httplib2. We're having issues with both container_pull and container_push though. Seems like the issue is in containerregistry, so I'll go bug them. :)

[sbueringer]

We're having the same problem with our registry with custom certs. That's our solution:

pip install httplib2.system-ca-certs-locater
# You may have to install the custom certificate in your OS truststore
# If your OS stores the certs in `/etc/ssl/certs/ca-certificates.crt` you're done
# If not, set the env var `HTTPLIB_CA_CERTS_PATH`, e.g.:
export HTTPLIB_CA_CERTS_PATH=/etc/ssl/certs/ca-bundle.crt

[roblg]

@sbueringer thanks for the suggestion. The pip install doesn't actually work for us, in part because httplib2.system-ca-certs-locater makes a pretty limited, specific assumption about where the default certificates file lives that for whatever reason doesn't match the way our environment is set up. I also see the HTTPLIB_CA_CERTS_PATH in the pypy page for the package, but I don't see anything in the code itself of the locater package, or of httplib2 core (and couldn't get it to make a difference) update: the code that's actually installed looks different than the github repo that's linked from the pypy page, so I dunno why it wasn't working. I'll take another crack at that, I guess.

I'm curious how folks in this thread feel about the proposal in #539? (assuming google/containerregistry#89 is merged first)

[sbueringer]

Yes I thought the same as I saw the code at the pypy page, but the installed code makes a lot more sense :).
So the difference between the different setups in our team is just the default location of the cert bundle in the OS. That's why Archlinux for example works per default, but Fedora needs the environment variable.

[nlopezgi]

Hi everyone,
we are working towards fixing this in #594 for push and will followup with a PR to fix for pull by enabling use of a custom docker client config.

[ali5h]

any update on this?

[nlopezgi]

it's implemented, but we have not updated the docs yet. You want to try it out? you need to import docker_toolchain_configure as here: https://github.com/bazelbuild/rules_docker/blob/master/testing/custom_toolchain_auth/WORKSPACE#L32. Note the client_config should point to the absolute path where the config is located (in the example we replace the value of client_config in the e2e tests here: https://github.com/bazelbuild/rules_docker/blob/master/testing/e2e.sh#L418.
And then import the custom container_pull as here https://github.com/bazelbuild/rules_docker/blob/master/testing/custom_toolchain_auth/WORKSPACE#L60 (note the name of the def there has 'local' just for testing purposes, but it does not need to use a local registry). (To use container_push with custom auth all you need is adding the docker_toolchain_rule to workspace). Do let me know if it works or you run into issues.

[ali5h]

For now, I fixed my problem by building a custom puller.par based on httplib2==0.12.0, and used that to override what rules_docker uses. This needs injecting proper env variable using action_env

[smukherj1]

We added some documentation on how to configure the docker toolchain rule to use custom authentication credentials specified via a docker client config.json in #686

You can specify a custom config.json when configuring the docker toolchain rule that will be used by every container_push rule automatically. container_pull requires a little extra work and our updated docs specifies what that is.

Note: All this does is make containerregistry load the config.json specified in the toolchain configuration. It's up to containerregistry to read what's in that file and interpret it appropriately.

Docker's official documentation on the client configuration file is at https://docs.docker.com/engine/reference/commandline/cli/#configuration-files

[sbueringer]

@smukherj1 I get that now it's possible to allow custom authentication credentials. But how can this be used to solve the problems with the certificates?

[smukherj1]

We had hoped specifying a custom client config would result in being able to specify a custom certificate using the instructions here. However, looking at the python containerregistry library's implementation, it doesn't look like it has the ability to load custom certifications specified in the docker client config directory. The python containerregistry uses the python http2 library to load certificates and using the environment variable seems to be the only way to make it use a custom certificate from rules_docker.

We are currently in the process of creating new versions of the container_pull & container_push rules that use the go-containerregistry library. Based on a cursory look, I'm not seeing any ability to use custom certificates there. It's possible that after rules docker completely switches to go-containerregistry, the environment variable method mentioned earlier in this issue will stop working. go-containerregistry might need to be modified to support using custom certificates by doing something like the accepted answer here.

[abergmeier]

@smukherj1 Funny you should mention go-containerregistry. I looked into it for working with kaniko (which in turn uses go-containerregistry). What is mostly used there is to (selectively) disable cert validation.
By now I think this is the saner option. And the latter is actually possible with go-containerregistry.

[smukherj1]

@abergmeier by the saner option were you referring to the ability to disable cert validation or the ability to use a custom certificate?

[neakor]

We're having the same problem with our registry with custom certs. That's our solution:

pip install httplib2.system-ca-certs-locater
# You may have to install the custom certificate in your OS truststore
# If your OS stores the certs in `/etc/ssl/certs/ca-certificates.crt` you're done
# If not, set the env var `HTTPLIB_CA_CERTS_PATH`, e.g.:
export HTTPLIB_CA_CERTS_PATH=/etc/ssl/certs/ca-bundle.crt

This solved our issue.

[AustinSchuhBRT]

This appears to be working for me now that rules_docker is using a golang based puller. (without any workarounds)

[nlopezgi]

Great! Let's close this then and if anyone experiences any other issues let me know to reopen.
For record: fixed as part of #580