Forbidden: Token does not provide access to channel #7671

hbrown-uiowa · 2023-10-10T20:53:49Z

Problem description

We updated our test instance to 2023.09 and did fresh registrations of Ubuntu 22.04, Ubuntu 20.04, and Rocky 8 to ORG 2

All three systems register via bootstrap.sh just fine to org 2, but all attempts to pull packages are met with 403 Forbidden

The rhn_web_ui.log has lots of
[ajp-nio-127.0.0.1-8009-exec-4] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: Token does not provide access to channel
followed by the channel name.

We can't patch any of the systems registered against it.

If I re-register one of the systems to org 1, then it can patch just fine.

Steps to reproduce

...

Uyuni version

Information for package Uyuni-Server-release:
---------------------------------------------
Repository     : Uyuni Server Stable
Name           : Uyuni-Server-release
Version        : 2023.09-230900.207.8.uyuni3
Arch           : x86_64
Vendor         : obs://build.opensuse.org/systemsmanagement:Uyuni
Support Level  : Level 3
Installed Size : 1.4 KiB
Installed      : Yes
Status         : up-to-date
Source package : Uyuni-Server-release-2023.09-230900.207.8.uyuni3.src
Summary        : Uyuni Server
Description    : 
    Uyuni lets you efficiently manage physical, virtual,
    and cloud-based Linux systems. It provides automated and cost-effective
    configuration and software management, asset management, and system
    provisioning.

Uyuni proxy version (if used)

No response

Useful logs

No response

Additional information

No response

The text was updated successfully, but these errors were encountered:

ppanon2022 · 2023-10-12T01:39:10Z

We're having the same problem. CentOS systems seem to still update fine after our Uyuni server update but Ubuntu systems are failing to download packages with the same token error on the server log. I tried to manually update (i.e. use salt cmd.run to download and apt install) the venv-salt-minion package on the client to the latest release because it was still on 3004-14.4.uyuni. However using 3006.0-15.1.uyuni didn't make a difference.

Is there any way this could have been broken by the changes for the Automated RHUI credential update? Seems like a long shot, but if it involves token maintenance, maybe it changed the tokens for Ubuntu systems somehow? Is there a shortcut for re-sync/regenerating tokens for Ubuntu clients that doesn't involve a re-registration?

hbrown-uiowa · 2023-10-13T15:01:47Z

@ppanon2022 we have run into this issue: #6842
and the solution was just to apply a highstate.

Unfortunately, applying the highstate doesn't fix it any more. I can only seem to get patches if I register against ORG 1.

I'm trying to replicate on a fresh install.

avshiliaev · 2023-10-13T16:49:54Z

Hey @hbrown-uiowa
Thanks for your report!
@lucidd does it seem like a known issue? I think there might've been something similar recently with the tokens.

hbrown-uiowa · 2023-10-13T20:00:17Z

I am able to replicate this on a fresh install.

Steps taken:

Install opensuse Leap 15.5
Follow process at https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/install-uyuni.html
Follow process at https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/uyuni-server-setup.html
create UYUNI as org 1
create testorg as org 2
set UYUNI to trust testorg
/usr/bin/spacewalk-common-channels -u uyuni-admin -a amd64-deb ubuntu-2204-pool-amd64-uyuni ubuntu-2204-amd64-main-uyuni ubuntu-2204-amd64-main-updates-uyuni ubuntu-2204-amd64-main-security-uyuni ubuntu-2204-amd64-universe-uyuni ubuntu-2204-amd64-universe-updates-uyuni ubuntu-2204-amd64-uyuni-client
Set all channels to public in UYUNI org
Create an activation key in testorg with key 2-ubuntu-2204

spacecmd {SSM:0}> activationkey_details 2-ubuntu-2204
Key:                    2-ubuntu-2204
Description:            ubuntu-22.04
Universal Default:      False
Usage Limit:            0
Deploy Config Channels: False
Contact Method:         default

Software Channels
-----------------
ubuntu-22.04-pool-amd64-uyuni
 |-- ubuntu-2204-amd64-main-security-uyuni
 |-- ubuntu-2204-amd64-main-updates-uyuni
 |-- ubuntu-2204-amd64-main-uyuni
 |-- ubuntu-2204-amd64-universe-updates-uyuni
 |-- ubuntu-2204-amd64-universe-uyuni
 |-- ubuntu-2204-amd64-uyuni-client

Configuration Channels
----------------------

Entitlements
------------


System Groups
-------------


Packages
--------

Generate the bootstrap script
Load an ubuntu 22.04 system, wget http:///pub/bootstrap/bootstrap.sh; edit script to include activation key above and run it, sign salt key, run salt-call state.highstate (which has lots of errors too). Run apt update and observe lots of 403 forbidden errors.

If I then repoint it to the the activation key for the UYUNI org (org 1), then it works fine.

ppanon2022 · 2023-10-13T20:33:34Z

@ppanon2022 we have run into this issue: #6842 and the solution was just to apply a highstate.

Unfortunately, applying the highstate doesn't fix it any more. I can only seem to get patches if I register against ORG 1.

I'm trying to replicate on a fresh install.

I think I may have seen that and did try to apply a high state already. Applying the high state works fine with no errors, but I still get the 403 failure.

The interesting thing is that we only have the one Org. Nothing fancy with trusts, but we still have that with systems joined to Org1. So there's no way for us to switch between orgs to a different org with a different activation key. I'm wondering if the fact that it's the last org in both our cases points to an off-by-one error on a loop or test somewhere. Because there seems to be a null org

# ls /var/spacewalk/packages/
1  NULL

maybe that's caused an issue for an undercount?

lucidd · 2023-10-17T08:10:47Z

@avshiliaev

Hey @hbrown-uiowa Thanks for your report! @lucidd does it seem like a known issue? I think there might've been something similar recently with the tokens.

what i can already say its a different issue from what i was working on given the error message and that it also happens on non ubuntu systems.

@hbrown-uiowa

I am able to replicate this on a fresh install.

Steps taken:

Install opensuse Leap 15.5

Follow process at https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/install-uyuni.html

Follow process at https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/uyuni-server-setup.html

create UYUNI as org 1

create testorg as org 2

set UYUNI to trust testorg

/usr/bin/spacewalk-common-channels -u uyuni-admin -a amd64-deb ubuntu-2204-pool-amd64-uyuni ubuntu-2204-amd64-main-uyuni ubuntu-2204-amd64-main-updates-uyuni ubuntu-2204-amd64-main-security-uyuni ubuntu-2204-amd64-universe-uyuni ubuntu-2204-amd64-universe-updates-uyuni ubuntu-2204-amd64-uyuni-client

Set all channels to public in UYUNI org

Create an activation key in testorg with key 2-ubuntu-2204
spacecmd {SSM:0}> activationkey_details 2-ubuntu-2204
Key:                    2-ubuntu-2204
Description:            ubuntu-22.04
Universal Default:      False
Usage Limit:            0
Deploy Config Channels: False
Contact Method:         default

Software Channels
-----------------
ubuntu-22.04-pool-amd64-uyuni
 |-- ubuntu-2204-amd64-main-security-uyuni
 |-- ubuntu-2204-amd64-main-updates-uyuni
 |-- ubuntu-2204-amd64-main-uyuni
 |-- ubuntu-2204-amd64-universe-updates-uyuni
 |-- ubuntu-2204-amd64-universe-uyuni
 |-- ubuntu-2204-amd64-uyuni-client

Configuration Channels
----------------------

Entitlements
------------


System Groups
-------------


Packages
--------
Generate the bootstrap script

Load an ubuntu 22.04 system, wget http:///pub/bootstrap/bootstrap.sh; edit script to include activation key above and run it, sign salt key, run salt-call state.highstate (which has lots of errors too). Run apt update and observe lots of 403 forbidden errors.

If I then repoint it to the the activation key for the UYUNI org (org 1), then it works fine.

does it only happen with the trusted org setup or is it also reproducible with just a simple single org setup? if not i assume there might be a problem related to trusted orgs.

hbrown-uiowa · 2023-10-17T13:57:46Z

I have assumed it is fine with just a single org (since registering to ORG 1 is fine both on our longstanding test instance and the one I spun up to replicate the issue).

Do you want me to try and test with just setting up ORG1 and nothing else?

For our production environment, collapsing the orgs is a non-starter. This is a blocker for us updating the production environment (which is limping along due to other bugs).

ppanon2022 · 2023-10-17T14:27:07Z

We are experiencing it with a single organization, although I don't have a way of reproducing it. When I was doing a spacewalk-data-fsck, I was getting a lot of errors about CentOS 7 aarch64 packages (from a product channel I had tried to delete) being in NULL instead of 1 (guessing based on the file path), so maybe that also triggered the bug or is related somehow. For example, a couple of lines from the spacewalk-data-fsck output: File path mismatch: /var/spacewalk/packages/NULL/5e6/hsqldb-demo/1:1.8.1.3-14.el7/noarch/5e6215650dac3faf700f8f408a6384eaff94a043040afcdf361721af6260a 220/hsqldb-demo-1.8.1.3-14.el7.noarch.rpm (org_id: 1 vs. NULL) File path mismatch: /var/spacewalk/packages/NULL/437/maven-jar-plugin-javadoc/2.4-8.el7/noarch/4378c2a6d7352cd15487ad009992da250c7615e9bfc1653204aa999 c87a40b15/maven-jar-plugin-javadoc-2.4-8.el7.noarch.rpm (org_id: 1 vs. NULL) If that is relevant then perhaps the issue is in the query that is pulling whatever token applies for that org somehow, although why it would apply to Ubuntu systems/channels and not CentOS is a mystery to me. But maybe the DB metadata for those packages or their rump product channel (that I can't seem to delete) is affecting a SQL query result in the same way that a second trusted channel does.

ppanon2022 · 2023-10-17T17:55:18Z

@avshiliaev You've probably already thought of this, but since I would expect this part of the code base to not need a lot of updating, there may be a fairly small number of code or schema change checkins since 2023.06 which could affect this scenario. Perhaps it might be useful to identify and review checkins to that functionality (token authentication and related references/dependencies) in the last 3 months? It definitely was working with 2023.06. I don't know what parts of the code base that might involve so I don't really know where to start on such a search for checkins.

If you find a query that you suspect it might be related to, I can certainly run that query on our database to return (sanitized) results.

hbrown-uiowa · 2023-10-17T19:24:42Z

It is working on 2023.04 (our production environment)

I re-ran my reproducer process but this time, I did not set any org trusts and I pulled in the ubuntu channels into org2. The bootstrap.sh didn't work because of apt key problems for an ubuntu 22.04 client, but after switching from venv-salt to regular salt, I was able to get bootstrapped and to patch.

It does appear to be something in the trusts (at least for me).

ppanon2022 · 2023-10-17T20:37:37Z

Because we have some packages appearing to be associated with a NULL org maybe there is some kind of equivalent implicit trust in our configuration/database as well. Perhaps something for that was changed in the rpm repo/minion handling code but without parallel changes in the deb repo/minion handling code. Or a mistake was made in a change to the deb/apt-related code that wasn't made in the rpm/yum handling code.

ppanon2022 · 2023-10-18T23:27:57Z

Should Uyuni even be expecting tokens for Ubuntu clients? It's been a few days since I was investigating, but I thought I remembered seeing that the token was kept in a client config file setting, and I couldn't find that setting on an Ubuntu minion. Which modules/methods on the server side handle the https calls and check for tokens?

ppanon2022 · 2023-10-19T00:49:03Z

Alright, the webui log error appears to be generated in DownloadController.java

           Opt.consume(orgClaim, () -> {
                LOG.info("Forbidden: Token does not specify the organization");
                halt(HttpStatus.SC_BAD_REQUEST, "Token does not specify the organization");
            }, orgId -> {
==>             if (!ChannelFactory.isAccessibleBy(channel, orgId)) {
                    LOG.info("Forbidden: Token does not provide access to channel {}", channel);
                    halt(HttpStatus.SC_FORBIDDEN, "Token does not provide access to channel " + channel);
                }

and that call in ChannelFactory runs a SQL Query

    /**
     * Checks if a channel is accessible by an Org.
     * @param channelLabel the channel label
     * @param orgId the Org ID
     * @return true if it is accessible
     */
    public static boolean isAccessibleBy(String channelLabel, Long orgId) {
        return (int)singleton.lookupObjectByNamedQuery("Channel.isAccessibleBy",
                Map.of("channel_label", channelLabel, ORG_ID, orgId)) > 0;
    }

and that query is

  <sql-query name="Channel.isAccessibleBy">
       <return-scalar type="int" column="result"/>
        <![CDATA[SELECT case when (EXISTS (
             SELECT 1
             FROM rhnChannel c
             JOIN rhnChannelFamilyMembers cfm ON cfm.channel_id = c.id
             JOIN rhnPrivateChannelFamily pcf ON pcf.channel_family_id = cfm.channel_family_id
             WHERE c.label = :channel_label
             AND pcf.org_id = :org_id
             LIMIT 1
       ) OR EXISTS (
             SELECT 1
             FROM rhnChannel c
             JOIN rhnChannelFamilyMembers cfm ON cfm.channel_id = c.id
             JOIN rhnPublicChannelFamily pcf ON pcf.channel_family_id = cfm.channel_family_id
             WHERE c.label = :channel_label
             LIMIT 1
       ) OR EXISTS (
             SELECT 1
             FROM rhnChannel c
             JOIN rhnTrustedOrgs tr ON c.org_id = tr.org_id
             WHERE c.parent_channel IS NULL AND c.channel_access = 'public'
             AND c.label = :channel_label
             AND tr.org_trust_id = :org_id
             LIMIT 1
       ) OR EXISTS (
             SELECT 1
             FROM rhnChannel c
             JOIN rhnChannelTrust tr ON c.id = tr.channel_id
             WHERE c.channel_access = 'protected'
             AND c.label = :channel_label
             AND tr.org_trust_id = :org_id
             LIMIT 1
       )) then 1 else 0 end AS result
       ]]>
   </sql-query>

I'll bet dollars to donuts that this commit is responsible.
cb6ed8b

So the next step is to run that query in our environment, substituting an Ubuntu channel label and the org_id and see what is being returned and why. Hbrown, can you get a chance to do that in your environment? The result is presumably 0, whereas one of those subselects should probably be returning a value. So our next step is to figure out what that data looks like, why it's not being matched for those Ubuntu channels but is for the ones that work, and how to fix it.

ppanon2022 · 2023-10-19T04:57:15Z

Oh, well that's how to solve hbrown's issue. You can compare the results of running the different subselects for working channels and broken ones against the uyuni database with psql to see what is failing to work. Assuming the proper associations are in place, that should make it possible to figure out what change in the query is needed. The subselect
SELECT 1
FROM rhnChannel c
JOIN rhnTrustedOrgs tr ON c.org_id = tr.org_id
WHERE c.parent_channel IS NULL AND c.channel_access = 'public'
AND c.label = :channel_label
AND tr.org_trust_id = :org_id
LIMIT 1
which involves the trusted orgs table seems to be likely the one that needs to work/be fixed. Or maybe you need to change the channel_access to public.

But after trying to find out what query parts work and what don't and finding no difference between query results for the different channels, I just realized our error is critically different.

Our error in rhn_web_ui.log is a close but different error message:
com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access

And that error appears to be because the client is not supplying a token in the request, failing the getTokenFromRequest in the DownloadPackage method in the same DownloadController.java module.

    if (checkTokens) {
        String token = getTokenFromRequest(request);
        validateToken(token, channelLabel, filename);
    }

so I've opened a discussion or a different ticket instead. @avshiliaev, if you or cbosdo could take a look at #7710 and answer some of my questions, I would really appreciate the help. At this point I don't know if it's a bug or a configuration issue, but I really need to find a solution ASAP.

Good luck with your problem hbrown.

mcalmer · 2023-10-19T05:44:02Z

@parlt91 this sounds like the issue we solved yesterday. Do you have already a PR for uyuni which can be linked here?

parlt91 · 2023-10-19T08:22:26Z

@parlt91 this sounds like the issue we solved yesterday. Do you have already a PR for uyuni which can be linked here?

PR to address the issue is here: #7711 we'll merge once we received positive feedback that it'll fix the issue.

parlt91 · 2023-10-19T09:55:38Z

PR is merged now

hbrown-uiowa · 2023-11-28T19:20:00Z

I believe this is working for us now after upgrading to 2023.10

marmau06 · 2023-11-30T13:04:01Z

Hello,
This morning, I updated Uyuni server to 2023.10, Debain do not work, again...
I upgraded venv-salt-minion to last version venv-salt-minion_3006.0-21.8.uyuni_amd64.deb on uyuni Debian 11 client. Do not work again...
I removed and re-registered the client, the same...

I still have this kind of message in rhn_web_ui.log:

2023-11-30 11:42:33,192 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-4] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/libxml2_2.9.10 dfsg-6.7 deb11u4.amd64-deb.deb
2023-11-30 11:42:33,202 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-6] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/bind9-dnsutils_1:9.16.44-1deb11u1.amd64-deb.deb
2023-11-30 11:42:33,206 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-9] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/bind9-libs_1:9.16.44-1deb11u1.amd64-deb.deb
2023-11-30 11:42:33,211 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-5] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/bind9-host_1:9.16.44-1deb11u1.amd64-deb.deb
2023-11-30 11:42:33,215 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-10] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/file_1:5.39-3 deb11u1.amd64-deb.deb
2023-11-30 11:42:33,220 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-7] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/libmagic1_1:5.39-3 deb11u1.amd64-deb.deb
2023-11-30 11:42:33,225 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-8] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/libmagic-mgc_1:5.39-3 deb11u1.amd64-deb.deb
2023-11-30 11:42:33,228 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-1] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/dnsutils_1:9.16.44-1deb11u1.all-deb.deb
2023-11-30 11:42:33,233 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-2] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/libaprutil1_1.6.1-5 deb11u1.amd64-deb.deb
2023-11-30 11:42:33,237 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-3] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/libapr1_1.7.0-6 deb11u2.amd64-deb.deb
2023-11-30 11:42:33,242 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-4] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/libnss3_2:3.61-1 deb11u3.amd64-deb.deb

Does any body fixed this issue with 2023.10? What did I forget ?

Thanks in advance, it's getting more urgent for us now, we are supposed to patch servers once a month... And we are already two monthes late....

marmau06 · 2023-11-30T13:23:59Z

If this can help, "wget" do not work on a simple package, we get 403 error...

wget https://myserver.france.mydomain:443/rhn/manager/download/debian-11-pool-amd64-uyuni/getPackage/shim-helpers-amd64-signed_1%2b15.7%2b1%7edeb11u1-X.amd64-deb.deb
--2023-11-30 14:19:48-- https://myserver.france.mydomain/rhn/manager/download/debian-11-pool-amd64-uyuni/getPackage/shim-helpers-amd64-signed_1%2b15.7%2b1%7edeb11u1-X.amd64-deb.deb
Resolving proxy.france.mydomain (proxymop.france.rexel)... 10.3.204.8
Connecting to proxy.france.mydomain (proxy.france.mydomain)|10.3.204.8|:8080... connected.
Proxy request sent, awaiting response... 403 403
2023-11-30 14:19:48 ERROR 403: 403.

But "curl" works fine on the same package...

curl -o /tmp/herve https://myserver.france.mydomain/rhn/manager/download/debian-11-pool-amd64-uyuni/getPackage/shim-helpers-amd64-signed_1%2b15.7%2b1%7edeb11u1-X.amd64-deb.deb
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4089 100 4089 0 0 159k 0 --:--:-- --:--:-- --:--:-- 159k

santeri3700 · 2023-11-30T15:38:45Z

There is a separate issue related to partially invalid repodata on DEB channels which causes similar token/403 errors. More details on #7788.

mcalmer · 2024-02-20T09:19:19Z

seems the original issue is solved and the second has its own issue. Closing

hbrown-uiowa added bug Something isn't working P5 labels Oct 10, 2023

avshiliaev added P2 and removed P5 labels Oct 13, 2023

ateel-dxs mentioned this issue Nov 6, 2023

Uyuni 2023-09 upgrade made Debian client failing with 403 error when accessing to Uyuni repositories #7788

Open

mcalmer closed this as completed Feb 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Forbidden: Token does not provide access to channel #7671

Forbidden: Token does not provide access to channel #7671

hbrown-uiowa commented Oct 10, 2023

ppanon2022 commented Oct 12, 2023 •

edited

Loading

hbrown-uiowa commented Oct 13, 2023

avshiliaev commented Oct 13, 2023 •

edited

Loading

hbrown-uiowa commented Oct 13, 2023

ppanon2022 commented Oct 13, 2023 •

edited

Loading

lucidd commented Oct 17, 2023

hbrown-uiowa commented Oct 17, 2023

ppanon2022 commented Oct 17, 2023 via email •

edited

Loading

ppanon2022 commented Oct 17, 2023 •

edited

Loading

hbrown-uiowa commented Oct 17, 2023

ppanon2022 commented Oct 17, 2023 •

edited

Loading

ppanon2022 commented Oct 18, 2023 •

edited

Loading

ppanon2022 commented Oct 19, 2023 •

edited

Loading

ppanon2022 commented Oct 19, 2023 •

edited

Loading

mcalmer commented Oct 19, 2023

parlt91 commented Oct 19, 2023

parlt91 commented Oct 19, 2023

hbrown-uiowa commented Nov 28, 2023

marmau06 commented Nov 30, 2023

marmau06 commented Nov 30, 2023

santeri3700 commented Nov 30, 2023

mcalmer commented Feb 20, 2024

Forbidden: Token does not provide access to channel #7671

Forbidden: Token does not provide access to channel #7671

Comments

hbrown-uiowa commented Oct 10, 2023

Problem description

Steps to reproduce

Uyuni version

Uyuni proxy version (if used)

Useful logs

Additional information

ppanon2022 commented Oct 12, 2023 • edited Loading

hbrown-uiowa commented Oct 13, 2023

avshiliaev commented Oct 13, 2023 • edited Loading

hbrown-uiowa commented Oct 13, 2023

ppanon2022 commented Oct 13, 2023 • edited Loading

lucidd commented Oct 17, 2023

hbrown-uiowa commented Oct 17, 2023

ppanon2022 commented Oct 17, 2023 via email • edited Loading

ppanon2022 commented Oct 17, 2023 • edited Loading

hbrown-uiowa commented Oct 17, 2023

ppanon2022 commented Oct 17, 2023 • edited Loading

ppanon2022 commented Oct 18, 2023 • edited Loading

ppanon2022 commented Oct 19, 2023 • edited Loading

ppanon2022 commented Oct 19, 2023 • edited Loading

mcalmer commented Oct 19, 2023

parlt91 commented Oct 19, 2023

parlt91 commented Oct 19, 2023

hbrown-uiowa commented Nov 28, 2023

marmau06 commented Nov 30, 2023

marmau06 commented Nov 30, 2023

santeri3700 commented Nov 30, 2023

mcalmer commented Feb 20, 2024

ppanon2022 commented Oct 12, 2023 •

edited

Loading

avshiliaev commented Oct 13, 2023 •

edited

Loading

ppanon2022 commented Oct 13, 2023 •

edited

Loading

ppanon2022 commented Oct 17, 2023 via email •

edited

Loading

ppanon2022 commented Oct 17, 2023 •

edited

Loading

ppanon2022 commented Oct 17, 2023 •

edited

Loading

ppanon2022 commented Oct 18, 2023 •

edited

Loading

ppanon2022 commented Oct 19, 2023 •

edited

Loading

ppanon2022 commented Oct 19, 2023 •

edited

Loading