Uyuni 2023-09 upgrade made Debian client failing with 403 error when accessing to Uyuni repositories #7788
Comments
|
Found this in /var/log/rhn/rhn_web_ui.log: 2023-11-02 14:23:23,379 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-4] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-10-amd64-main-security-uyuni/getPackage/libx11-data_2:1.6.7-1 deb10u4.all-deb.deb |
|
Same problem on debian 10. Patching in general fails: Err:1 https://uyuni.:443/rhn/manager/download debian_10-debian_10_dev_qa-debian-10-amd64-main-security-uyuni/ python3.7-dev 3.7.3-2+deb10u6 |
|
This looks related to #7671 which has been resolved but needs a release. For the record, I have Ubuntu systems that are all having the exact same issue. I get a 403 from apt when it attempts to retrieve packages and that forbidden access token message is present in my logs on 2023.09 as well. |
|
Hey everyone, thanks for reporting it. It's worth investigating in general, but I would start here by trying to apply the highstate on the client to see if it solves the token issue. |
I just trying to set the highstate - nothing changed. Still the same error. But how should it change - it worked before. |
|
By chance, before the 403 error, did any one of you changed software channels or linked new software channels to already existing repositories? I did and encountered the 403 afterwards on my Ubuntu machines (the clients still try to download packages from now defunct paths). RedHat based distros do not suffer from the same fate. Still trying to figure a way out of this mess... So far I tried:
Still to be tried: deleting everything once more and creating the repositories with different names, maybe that will do it |
|
Hello, I already put this in (#7671) Hello, I still have this kind of message in rhn_web_ui.log: 2023-11-30 11:42:33,192 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-4] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/libxml2_2.9.10 dfsg-6.7 deb11u4.amd64-deb.deb Does any body fixed this issue with 2023.10? What did I forget ? Thanks in advance, it's getting more urgent for us now, we are supposed to patch servers once a month... And we are already two monthes late.... |
|
If this can help, "wget" do not work on a simple package, we get 403 error... wget https://myserver.france.mydomain:443/rhn/manager/download/debian-11-pool-amd64-uyuni/getPackage/shim-helpers-amd64-signed_1%2b15.7%2b1%7edeb11u1-X.amd64-deb.deb But "curl" works fine on the same package... curl -o /tmp/herve https://myserver.france.mydomain/rhn/manager/download/debian-11-pool-amd64-uyuni/getPackage/shim-helpers-amd64-signed_1%2b15.7%2b1%7edeb11u1-X.amd64-deb.deb |
|
Looks like Taskomatic is generating partially invalid repodata for DEB channels/repos. Some of the packages in the Packages file are pointing to a wrong channel and clients will get Error 403 unless they are subscribed to the wrong channel as well. Example (see the difference between the directory channel label and the Filename channel label): I've confirmed this on two separate Uyuni Servers running 2023.09 and 2023.10 with Ubuntu 20.04 and 22.04 clients. Validating repodataI wrote a small Python 3 script to list all of the channels and packages which have invalid repodata. I've successfully confirmed that the packages listed by my script will fail to download on DEB based clients which are NOT subscribed to the channels which the repodata is incorrectly pointing to.
WorkaroundFor just a few independent clients facing this issue, fixing the channel labels from the apt cache works although the issue will return on the next package list refresh / apt update. A global workaround could be to temporarily disable the repo-sync schedules and the "channel-repodata-default" task schedule from Taskomatic and manually fixing the channel labels from the Packages files on the Uyuni Server ( Possible causeI suspect that this issue was possibly caused by commit 72c8881 which was merged before 2023.09 release with #7566. |
|
Thanks @santeri3700, I can confirm that your local workaround (editing the "*_Packages" file) works. Unfortunately on my installation your python script reveals over 150.000 packages which are pointing to wrong channels. Until Taskomatic is fixed I guess I need to build a second script that (on the client side) loops over every "Filename" entry in "*_Packages" and changes it to the actual name of the channel... |
|
Good morning, we found the same issue on Ubuntu 20.04 in ubuntu-2004-amd64-uyuni-client repository. Error message: Steps to reproduce
Uyuni version Uyuni proxy version (if used): |
|
Also having this issue on 2023.09. |
|
Hi All, Not sure of translation from french to english, but it felt in running mode :-) ..... |
|
Looks like this issue has partially resolved itself on my environments. There are still some channels with incorrect Packages files and they seem to have not been updated for a very long time (e.g. last modified time was mid Nov 2023) because some of the upstream repositories haven't been updated for a while. I double checked and repo-syncs are still happening but it seems Uyuni might not be executing the ChannelRepodataWorker and/or DebRepositoryWriter on those channels. Maybe this is because there are no new packages to be synchronized (since Nov 14 in this case)? Repodata staleness check seems to be implemented here:
So I tried updating the timestamp of the Packages.gz file: But DebRepositoryWriter still does not seem to run after the timestamp modification or after trying to do the following:
|
|
I found out a way to force repodata regeneration and created another script to make it more convenient to automatically fix DEB channels with invalid repodata (without the previously mentioned workaround). I tested this script with two Uyuni 2023.10 servers and it fixed all DEB channels with invalid repodata after Taskomatic was done running the 'channel-repodata' task(s). I'm still not sure about the reason why this invalid data came to be. Perhaps it was not directly caused by the pull request I mentioned earlier. Invalid DEB repodata regeneration script
|
|
This problem hasn't reappeared in my environments after the repodata regen and upgrade to Uyuni 2023.12 so I think this issue could be closed. |
|
I don't think this issue should be closed. Your work and research notwithstanding, you almost single-handedly kept this issue alive and delivered an after the fact Third-Party-Fix (though I have not tested it) - but this can only be a band aid in my opinion. What interests me is how this problem could occur in the first place? I "solved" the problem by wiping the whole database and building everything from the ground up (I'm on Uyuni Server 2023.09 right now). Solved is in quotes because even after setting up everything anew a lot of packages are still in multiple channels (Ubuntu 22.04 LTS). This works because all the channels are subscribed to but this is still very much ugly. How can this be? |
|
@cFabij can you update your uyuni server to the latest version. After that re-run repo-sync, and if you are using CLM, then make any change to the project then build promote (this will force metadata re-generate) |
|
@rjmateus I updated to 2024.01 right now and run Afterwards I had over 8k packages in no channels. I deleted those. I still have packages in multiple channels though (e.g.): But I think there are less of those instances now. |
|
@cFabij that is probably intentional for some of the packages you see belonging to multiple channels at once. If you inspect the Ubuntu upstream repositories you can see that the specific package is present in both Main Updates and Universe repos. Same package in multiple channels shouldn't normally be an issue. |
Good morning, I agree that normally there is no problem to find packages in multiple channels. In this case we don´t understand why the venv-salt-minion_3006.0 packages are located to the ubuntu-2004-amd64-uyuni-client-devel repository. In November 2023 there was no problem and the only change was the installation of release 2023.10-230900.209.1.uyuni3. Kind regards |
|
@Ripper66Six not sure if I understand you correctly but it seems that it is the same case as with the previously mentioned packages and repos. Some packages happen to be identical (see checksums) in both repositories and that's why it may seem that the packages are coming from the wrong channel/repository. ubuntu-2004-amd64-uyuni-client (Stable): ubuntu-2004-amd64-uyuni-client-devel (Master): I personally don't even use or sync the devel client tools repos as I don't have any need for them. The stable client tools repos should be sufficient for non-development environments. |
|
I just ran into what looks to be the same issue 2024.05 after upgrading a host from Rocky Linux 8 to Rocky Linux 9. It was working on the upgraded hosts then suddenly stopped and the server logs contain |
Problem description
Hello, since we upgraded Uyuni server to 2023-09 level, our Debian clients can not access to Uyuni repositories, failing with 403 (forbidden) error code.
Nothing has changed in our infrastructure...
Still investigating, but may be the solution will be an evidence for any of you ? :)
Steps to reproduce
2.apt-get dist-upgrade on debian client...
...
Uyuni version
Uyuni proxy version (if used)
No response
Useful logs
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: