Self-hosted Zezere doesn't receive a request from ignition

[Rintsi]

I'm running self-hosted Zezere with minimum settings and everything seems to work fine (I can add SSH keys etc.). I have a Fedora IoT running on Raspberry PI 4 (I know it is not officially supported, but I was able to claim the device with the official provisioning service, so I think it should work) and I change the Zezere URL by adding the configuration option to kernel command line (the support was added here: f66c0b6). By examining the logs while running self-hosted Zezere I cannot see any requests coming in from the device during ignition. I was able to confirm that it should use the newly configure provisioning server, as it is shown correctly in the MOTD.

Not sure where to go from here

[puiterwijk]

Hi,

Thanks for your question.
If the banner does print the correct URL, it should be actually using that.

Some things you could check are:

• Is the URL you put in without any trailing bits? (i.e. http://myserver.com:9090, instead of http://myserver.com:9090/netboot/config?
• Is the URL you put in available from the system? i.e. the DNS name can be resolved with the name servers from DHCP and the port is open in the firewall?
• If you're using a proxy to offer TLS, is it working correctly, and can you see the requests on there?
• Is the URL you put in available without TLS trust root changes. i.e. is it either using http:// (non-TLS), or is it using a certificate issued by a CA that's in the standard trust list. (and is it sending the correct intermediates to complete the trust chain?)

Which logs are you following? If you use Zezere as the shipped container (docker run quay.io/fedora-iot/zezere), it should be printing out all the requests that it receives.
In that case, I'd be curious about your container setup, i.e. is the port open from the Zezere client?

If you're not using the container, but using it as a wsgi application, how are you deploying it?

[Rintsi]

Hi!

• It is without any trailing bits.
• I have a static public IP so I actually have the name on public DNS and route it to my computer that was publicly available
• Without TLS, I can also try to generate Let's Encrypt certificate if it is needed

I'm running Zezere by running with wsgi directly from the development machine command line. I might be missing some understanding how the whole thing should work. The static directory is missing from the Zezere repository and thus I don't have the netboot config files. Are those needed before the request for ignition comes in?

I can also try with running the container

[puiterwijk]

So, TLS is not needed, but just figured to mention it as a possible point.
To ensure: you did verify that you can reach the Zezere URL from a system on the same network?
I'm not sure whether the devel server prints out requests that occur.

Are you sure that the requests don't come in, or is it just because you don't see hte logs?

[Rintsi]

I didn't have time to go through all the code in order to figure out what really happens during ignition. I concluded that the requests wouldn't be coming in because of two reasons:

1. I cannot see any requests in the (django) logs (while I can see GET requests to endpoints)
2. Zezere doesn't show the device in the "Claim Unowned Devices"

Also, as stated before, I already claimed the device once with the official Zezere. Is there a state for that in the device? Should I retry this with a fresh installation of Fedora IoT?

Clarification for number 2. Both Fedora IoT device and the computer used to access Zezere are coming from the same public IP (my ISP). The Zezere development environment is connected via different ISP mobile network that has a static public IP and the development machine is connected to that. (I undestood that in order to claim the device the requests need to come from the same public IP)

[jsimmonds2]

Maybe same :

Using latest Fedora IoT ( Linux fedora 5.12.11-300.fc34.aarch64 #1 SMP Wed Jun 16 )
my Raspberry Pi 3 Model B Plus always fails to register with Zezere and the Console login message always
includes "... (device ID not yet known)"

Journalctl says

Jun 24 18:26:16 fedora systemd[1]: Starting Run Ignition for Zezere...
Jun 24 18:26:18 fedora zezere-ignition[935]: INFO     : Ignition 2.10.1
Jun 24 18:26:18 fedora zezere-ignition[935]: INFO     : Stage: fetch
Jun 24 18:26:18 fedora zezere-ignition[935]: INFO     : no config dir at "/usr/lib/ignition/base.d"
Jun 24 18:26:18 fedora zezere-ignition[935]: INFO     : no config dir at "/usr/lib/ignition/base.platform.d/file"
Jun 24 18:26:18 fedora zezere-ignition[935]: DEBUG    : parsed url from cmdline: ""
Jun 24 18:26:18 fedora zezere-ignition[935]: INFO     : no config URL provided
Jun 24 18:26:18 fedora zezere-ignition[935]: INFO     : reading system config file "/usr/lib/ignition/user.ign"
Jun 24 18:26:18 fedora zezere-ignition[935]: INFO     : no config at "/usr/lib/ignition/user.ign"
Jun 24 18:26:18 fedora zezere-ignition[935]: INFO     : using config file at "/tmp/zezere-ignition-config-kcpszj7j.ign"
Jun 24 18:26:18 fedora zezere-ignition[935]: DEBUG    : parsing config with SHA512: 21e0d3c2ff71e85f63512d2ca9f0bf1d4dcb59881ce8f190d7004014dc0b0cd2c1b4f011bf4562a86b621>
Jun 24 18:26:18 fedora zezere-ignition[935]: INFO     : GET https://provision.fedoraproject.org/netboot/aarch64/ignition/b8:27:eb:d7:f3:90: attempt #1
Jun 24 18:26:19 fedora zezere-ignition[935]: INFO     : GET result: Not Found
Jun 24 18:26:19 fedora zezere-ignition[935]: WARNING  : failed to fetch config: resource not found
Jun 24 18:26:19 fedora zezere-ignition[935]: CRITICAL : failed to acquire config: resource not found
Jun 24 18:26:19 fedora zezere-ignition[935]: CRITICAL : Ignition failed: resource not found
Jun 24 18:26:19 fedora zezere-ignition[942]: INFO     : Ignition 2.10.1
Jun 24 18:26:19 fedora zezere-ignition[942]: INFO     : Stage: disks
Jun 24 18:26:19 fedora zezere-ignition[942]: INFO     : no config dir at "/usr/lib/ignition/base.d"
Jun 24 18:26:19 fedora zezere-ignition[942]: INFO     : no config dir at "/usr/lib/ignition/base.platform.d/file"
Jun 24 18:26:19 fedora zezere-ignition[942]: CRITICAL : failed to acquire config: open /run/ignition.json: no such file or directory
Jun 24 18:26:19 fedora zezere-ignition[942]: CRITICAL : Ignition failed: open /run/ignition.json: no such file or directory
Jun 24 18:26:19 fedora zezere-ignition[948]: INFO     : Ignition 2.10.1
Jun 24 18:26:19 fedora zezere-ignition[948]: INFO     : Stage: mount
Jun 24 18:26:19 fedora zezere-ignition[948]: INFO     : no config dir at "/usr/lib/ignition/base.d"
Jun 24 18:26:19 fedora zezere-ignition[948]: INFO     : no config dir at "/usr/lib/ignition/base.platform.d/file"
Jun 24 18:26:19 fedora zezere-ignition[948]: CRITICAL : failed to acquire config: open /run/ignition.json: no such file or directory
Jun 24 18:26:19 fedora zezere-ignition[948]: CRITICAL : Ignition failed: open /run/ignition.json: no such file or directory
Jun 24 18:26:19 fedora zezere-ignition[954]: INFO     : Ignition 2.10.1
Jun 24 18:26:19 fedora zezere-ignition[954]: INFO     : Stage: files
Jun 24 18:26:19 fedora zezere-ignition[954]: INFO     : no config dir at "/usr/lib/ignition/base.d"
Jun 24 18:26:19 fedora zezere-ignition[954]: INFO     : no config dir at "/usr/lib/ignition/base.platform.d/file"
Jun 24 18:26:19 fedora zezere-ignition[954]: CRITICAL : failed to acquire config: open /run/ignition.json: no such file or directory
Jun 24 18:26:19 fedora zezere-ignition[954]: CRITICAL : Ignition failed: open /run/ignition.json: no such file or directory
Jun 24 18:26:20 fedora zezere-ignition[960]: INFO     : Ignition 2.10.1
Jun 24 18:26:20 fedora zezere-ignition[960]: INFO     : Stage: umount
Jun 24 18:26:20 fedora zezere-ignition[960]: INFO     : no config dir at "/usr/lib/ignition/base.d"
Jun 24 18:26:20 fedora zezere-ignition[960]: INFO     : no config dir at "/usr/lib/ignition/base.platform.d/file"
Jun 24 18:26:20 fedora zezere-ignition[960]: CRITICAL : failed to acquire config: open /run/ignition.json: no such file or directory
Jun 24 18:26:20 fedora zezere-ignition[960]: CRITICAL : Ignition failed: open /run/ignition.json: no such file or directory
Jun 24 18:26:20 fedora zezere-ignition[934]: Running stage fetch with config file /tmp/zezere-ignition-config-kcpszj7j.ign
Jun 24 18:26:20 fedora zezere-ignition[934]: Running stage disks with config file /tmp/zezere-ignition-config-kcpszj7j.ign
Jun 24 18:26:20 fedora zezere-ignition[934]: Running stage mount with config file /tmp/zezere-ignition-config-kcpszj7j.ign
Jun 24 18:26:20 fedora zezere-ignition[934]: Running stage files with config file /tmp/zezere-ignition-config-kcpszj7j.ign
Jun 24 18:26:20 fedora zezere-ignition[934]: Running stage umount with config file /tmp/zezere-ignition-config-kcpszj7j.ign
Jun 24 18:26:20 fedora systemd[1]: zezere_ignition.service: Deactivated successfully.
Jun 24 18:26:20 fedora systemd[1]: Finished Run Ignition for Zezere.
Jun 24 18:26:20 fedora systemd[1]: zezere_ignition.service: Consumed 2.017s CPU time.