Our Ubuntu minions are unable to download packages during an apt upgrade after upgrading to 2023.09

[ppanon2022]

Because they were recent issues with Ubuntu after upgrading to 2023.09, at first I thought we had the same problem as #7671 because the error message was somewhat similar. However with further investigation we're having a different problem: our Ubuntu Uyuni clients don't appear to be supplying a token. The error in /var/log/rhn/rhn_web_ui.log is actually

2023-10-11 17:23:57,160 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-2] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access

That error appears to be because the client is not supplying a token in the request, failing the getTokenFromRequest in the DownloadPackage method in the DownloadController.java module.

        if (checkTokens) {
            String token = getTokenFromRequest(request);
            validateToken(token, channelLabel, filename);
        }

where that call tries

    /**
     * Return the token from the request or send an appropriate response if it is not there.
     *
     * @param request the request object
     * @return the token
     */
    private static String getTokenFromRequest(Request request) {
        Set<String> queryParams = request.queryParams();
        String header = request.headers("X-Mgr-Auth");
        header = StringUtils.isNotBlank(header) ? header : getTokenForDebian(request);
        if (queryParams.isEmpty() && StringUtils.isBlank(header)) {
            if (LOG.isInfoEnabled()) {
                LOG.info("Forbidden: You need a token to access {}", request.pathInfo().replaceAll("[\n\r\t]", "_"));
            }
            halt(HttpStatus.SC_FORBIDDEN, String.format("You need a token to access %s", request.pathInfo()));
        }
        if ((queryParams.size() > 1 && header == null) ||
                (!queryParams.isEmpty() && header != null)) {
            LOG.info("Bad Request: Only one token is accepted");
            halt(HttpStatus.SC_BAD_REQUEST, "Only one token is accepted");
        }
        if (!queryParams.isEmpty()) {
            return queryParams.iterator().next();
        }
        else {
            return header;
        }
    }

So my questions are: Why wouldn't the client be putting the token in the request header? Where does the Uyuni client/minion store the token if it's available? and assuming that location is empty on those clients somehow, is there a way to push a token to the client with salt to avoid having to re-register the client (assuming that works).

[ppanon2022]

P.S. I've tried upgrading the venv-salt-minion package on one of our test clients to venv-salt-minion-3006.0-15.1.uyuni.amd64-deb.deb (with a manual download of the file followed by an apt install of the local file, in case the older version was a factor, but it didn't make a difference.

[ppanon2022]

Are the tokens what's in /etc/apt/auth.conf.d/susemanager.conf? The entries in there seem to be channel specific, which seems to match up with some of the later comparisons in DownloadController. But if those entries are the tokens then why wouldn't they be passed on, and is there a salt formula that can be used to refresh them since the header says it's managed by SUSE Manager? Renaming that file causes 403 errors trying to get Uyuni channel metadata so it seems likely that those are tokens, but if so why are they submitted for apt update but not apt upgrade?

[lucidd]

this sounds a lot like its the same issue i recently fixed #7566. Is this happening on cloned channels? (incudes CLM). If so the problem is that the metadata is generated from faulty cached snippets that are channel specific so the path for a package apt requests does not match the path the tokens are configured for so apt won't sent the token for those requests.
You can work around this issue until the fix is released by disabling token checks java.salt_check_download_tokens = false or disabling the per package metadata caching user_db_repodata = false.

[ppanon2022]

Yes, we use LCM and the systems we were testing with were using LCM project-created channels. I then tried switching one of the systems to a non-LCM channel group and that made no difference. I will try the workaround you suggest to see if it works and let you know. Thank you very much for your prompt reply.

[ppanon2022]

Can confirm using java.salt_check_download_tokens = false as a workaround allowed the updates to proceed on one of our test clients. What are the tradeoffs of using one workaround vs. another? Is java.salt_check_download_tokens less secure and caching user_db_repodata = more performance impacting?

[lucidd]

java.salt_check_download_tokens disables token checks on the download endpoint so there are security implications if you only want trusted onboarded clients to have access to channel data. No checking also means less overhead so it might have some small performance benefit.
user_db_repodata disables the per package metadata caching when generating the channel metadata. So generating channel metadata after a sync or clone will be slower (if i remember correctly in some tests it was around 3 seconds vs 30). So while it sounds like a big improvement for the occasional sync/clone waiting 30 seconds shouldn't make much of a difference.

[ppanon2022]

Thank you for the clarification!

[ppanon2022]

I have another issue and I don't know if I should open a new discussion or not. I was able to successfully update our traditional Uyuni Proxies without too much trouble, and they work to update clients associated with those proxies.

However I also needed to create a new Uyuni proxy, and I used openSUSE 15.5 and the Uyuni Proxy install for 2023.09. That proxy almost works but fails on the key operation of updating clients. I set up a new test Ubuntu client and registered it with the proxy. Everything looks fine. I can run salt commands to the client through the new proxy such as a state.apply. I can do an apt update from the client and it fetches the repodata information with no problem. But when I try to do an apt upgrade to actually update packages on that client system, it lists the appropriate packages that need to be upgraded, but it fails to download them. When I try to do it through the Uyuni console, the error is similar. The entries in the venv-salt-minion.log on the client show

Ign:1 https://{new_proxy_fqdn}:443/rhn/manager/download ubuntu22-prod-ubuntu-2204-amd64-main-updates-uyuni/ linux-firmware 20220329.git681281e4-0ubuntu3.19
Ign:2 https://{new_proxy_fqdn}:443/rhn/manager/download ubuntu22-prod-ubuntu-2204-amd64-uyuni-client/ venv-salt-minion 3006.0-15.1.uyuni
Ign:1 https://{new_proxy_fqdn}:443/rhn/manager/download ubuntu22-prod-ubuntu-2204-amd64-main-updates-uyuni/ linux-firmware 20220329.git681281e4-0ubuntu3.19
Get:2 https://{new_proxy_fqdn}:443/rhn/manager/download ubuntu22-prod-ubuntu-2204-amd64-uyuni-client/ venv-salt-minion 3006.0-15.1.uyuni [23.0 MB]
Ign:1 https://{new_proxy_fqdn}:443/rhn/manager/download ubuntu22-prod-ubuntu-2204-amd64-main-updates-uyuni/ linux-firmware 20220329.git681281e4-0ubuntu3.19
Err:1 https://{new_proxy_fqdn}:443/rhn/manager/download ubuntu22-prod-ubuntu-2204-amd64-main-updates-uyuni/ linux-firmware 20220329.git681281e4-0ubuntu3.19
Connection failed [IP: {IP address of proxy}]
Fetched 23.0 MB in 5min 7s (74.9 kB/s)
2023-11-02 07:59:46,005 [salt.loaded.int.module.cmdmod:884 ][ERROR ][22841] stderr: Running scope as unit: run-ree0a7c18f9f4446eb
2c94f4b42c8fcd3.scope
E: Failed to fetch https://{new_proxy_fqdn}:443/rhn/manager/download/ubuntu-2204-amd64-main-updates-uyuni/getPackage/linux-firmware_20220329.git681281e4-0ubuntu3.19.all-deb.deb Connection failed [IP: {IP address of proxy}]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
2023-11-02 07:59:46,005 [salt.loaded.int.module.cmdmod:886 ][ERROR ][22841] retcode: 100
2023-11-02 07:59:46,071 [salt.state :319 ][ERROR ][22841] Problem encountered installing package(s). Additional info follow
s:

errors:
- Running scope as unit: run-ree0a7c18f9f4446eb2c94f4b42c8fcd3.scope
E: Failed to fetch https://<new_proxy_fqdn>:443/rhn/manager/download/ubuntu-2204-amd64-main-updates-uyuni
/getPackage/linux-firmware_20220329.git681281e4-0ubuntu3.19.all-deb.deb Connection failed [IP: {IP address of proxy}]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

When I run it interactively with apt upgrade it seems to try to get the headers but fail afterwards, and the log above seems to indicate it might be OK downloading the new venv-salt-minion package but choking on the kernel package. The message I see when it's trying to get the package is

59% [Waiting for headers]
At one point I see a throughput value on the right of the same line (36MB/s to 40Mb/s) but it's short lived and goes away.

The entries in the squid log are
1698951748.540 120103 127.0.0.1 TCP_MISS_ABORTED/000 0 GET http://<new_proxy_fqdn>l/rhn/manager/download/ubuntu-2204-amd64-main-updates-uyuni/getPackage/linux-firmware_20220329.git681281e4-0ubuntu3.19.all-deb.deb - HIER_DIRECT/{Proxy IP}

Any suggestions?

[ppanon2022]

I was able to specifically install a new version of the venv-salt-minion package with apt upgrade venv-salt-minion, so it appears to be specific to downloading the kernel package. Could it be a timeout of some kind due to the network lag between this remote proxy and the main Uyuni server. Is there something I can use to try to increase the timeout? I've tried to add commit_timeout = 5 minutes in squid.donf with no effect.

The problem may actually be in the python wsgi handler. It gets the following errors in the apache2/error_log:

[Fri Nov 03 03:02:28.539146 2023] [wsgi:error] [pid 21449] [client {proxy_ip_address}:39002] URI: /rhn/manager/download/ubuntu-2204-amd64-main-updates-uyuni/getPackage/linux-firmware_20220329.git681281e4-0ubuntu3.19.all-deb.deb
[Fri Nov 03 03:02:28.539257 2023] [wsgi:error] [pid 21449] [client {proxy_ip_address}:39002] \tPATH_INFO: /ubuntu-2204-amd64-main-updates-uyuni/getPackage/linux-firmware_20220329.git681281e4-0ubuntu3.19.all-deb.deb
[Fri Nov 03 03:02:28.539267 2023] [wsgi:error] [pid 21449] [client {proxy_ip_address}:39002] \tPATH_TRANSLATED: /srv/www/htdocs/ubuntu-2204-amd64-main-updates-uyuni/getPackage/linux-firmware_20220329.git681281e4-0ubuntu3.19.all-deb.deb
[Fri Nov 03 03:02:28.539327 2023] [wsgi:error] [pid 21449] [client {proxy_ip_address}:39002] \tREQUEST_URI: /rhn/manager/download/ubuntu-2204-amd64-main-updates-uyuni/getPackage/linux-firmware_20220329.git681281e4-0ubuntu3.19.all-deb.deb
[Fri Nov 03 03:02:28.539358 2023] [wsgi:error] [pid 21449] [client {proxy_ip_address}:39002] \tSCRIPT_URI: https://{new_proxy_fqdn}/rhn/manager/download/ubuntu-2204-amd64-main-updates-uyuni/getPackage/linux-firmware_20220329.git681281e4-0ubuntu3.19.all-deb.deb
[Fri Nov 03 03:02:28.539369 2023] [wsgi:error] [pid 21449] [client {proxy_ip_address}:39002] \tSCRIPT_URL: /rhn/manager/download/ubuntu-2204-amd64-main-updates-uyuni/getPackage/linux-firmware_20220329.git681281e4-0ubuntu3.19.all-deb.deb

but that doesn't really help identify what the error is.
rhn_proxy_broker.log has
2023/11/02 15:54:34 +08:00 14232 wsgi_proxy.broker {proxy_ip_address}: broker/rhnBroker.handler('Leaving handler with status code 304',)
2023/11/02 15:54:35 +08:00 14232 wsgi_proxy.broker {proxy_ip_address}: broker/rhnBroker.handler('Leaving handler with status code 304',)
2023/11/02 15:54:37 +08:00 14232 wsgi_proxy.broker {proxy_ip_address}: broker/rhnBroker.handler('Leaving handler with status code 304',)

But why would I be getting a status not modified error if I don't already have the file?

[ppanon2022]

I tried to delete the system from Uyuni and then re-register it with the bootstrap script. I then reran the apt-get update followed by apt-get upgrade. At first I thought it was working because it started downloading dozens of outstanding packages, until it got to the same linux-firmware package. So it appears to be something specific to that linux-firmware package. Should I delete the package so that it can be re-sync'ed from the Ubuntu repos? I'm at a loss of how we got here or what to do to fix it.

[ppanon2022]

I wound up manually downloading the linux-firmware package from the main Uyuni server and installing it with dpkg, then the rest upgraded fine with apt update && apt upgrade. So maybe it's the fact that that package seems to be one of the largest ones at 247MB. In addition to increasing Squid's connect timeout value, I also am trying to see if increasing client_request_buffer_max_size as suggested by https://bugs.squid-cache.org/show_bug.cgi?id=5214 prevents a repeat later.