Orin NX 16gb on Orin Nano DevKit has issues booting via USB on firmware 35.3.1

[chansen-and]

I found that when I updated the Jetpack Firmware to 35.3.1 using nix build .#initrd-flash-orin-nx-devkit I lose the ability to boot via USB in the UEFI menu, it does not seem to detect the device.

Flashing back to 35.2.1 seems to fix this.

I did some additional UEFI troubleshooting just to confirm it was the UEFI being built by jetpack-nixos that was the issue:

Test #1:

• In the generated initrd script (initrd-flash-orin-nx-devki) I commented out this line:

  jetpack-nixos/flash-script.nix

  Line 50 in ddaff1b

  cp ${uefi-firmware}/uefi_jetson.bin bootloader/uefi_jetson.bin

  in order to use the mainstream 35.3.1 UEFI image that is normally written over (Jetson UEFI firmware (version 3.1-32827747 built on 2023-03-19).

• Then, during flashing before it could boot the Linux initrd to update I hit ESC to get to the UEFI and confirmed that the UEFI could boot from the USB. However, because this was copied in after it was built it seems this version is only used for the initrd boot and is not programmed in.

Test #2

• I modified uefi_firmware.nix to use the old 35.2.1 branches that were referenced in commit f55f179.
• With these changes I was able to flash and boot from a USB device.
  See https://github.com/chansen-and/jetpack-nixos/tree/chansen/orin-nx-usb-boot-fix or exact changes used to test.

[danielfullmer]

We were actually unable to reproduce this issue on our Orin NX or Orin Nano. I just merged a PR that includes an update to the usb controller firmware that NVIDIA includes, which may help. Would you be able to reflash and test again with firmware after 48d86a9 ?

[danielfullmer]

Just checking in again if you were able to reproduce this issue with the latest commit?

[chansen-and]

@danielfullmer Yep, I tested it and still am seeing the same issue.

TLDR for those who stumble upon this issue:

• When I have an SSD Plugged in I am able to see USB as a bootable option.
• When I remove the SSD I no longer see the USB as a bootable option.

I'll test again with 35.4.1 at some point.

[emrahbillur]

I'm facing the same issue with Orin Nano Developer Kit, tested with both Orin NX 16 GB and Orin Nano. Same issue faced.

Could not locate device tree fixup protocol, skipping.
EFI stub: Booting Linux Kernel...
EFI stub: ERROR: Invalid header detected on UEFI supplied FDT, ignoring ...
EFI stub: Generating empty DTB
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Exiting boot services and installing virtual address map...

Tested with even overlays with different device tree configurations for orin nano developer kit. Firmware boots original NVIDIA images from nvme but any nixos image on usb fail.

[mikatammi]

I'm facing the same issue with Orin Nano Developer Kit, tested with both Orin NX 16 GB and Orin Nano. Same issue faced.

Could not locate device tree fixup protocol, skipping.
EFI stub: Booting Linux Kernel...
EFI stub: ERROR: Invalid header detected on UEFI supplied FDT, ignoring ...
EFI stub: Generating empty DTB
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Exiting boot services and installing virtual address map...

Tested with even overlays with different device tree configurations for orin nano developer kit. Firmware boots original NVIDIA images from nvme but any nixos image on usb fail.

This is probably due to fact systemd-boot does not allocate any extra space for the device tree. It can be patched for example like this (a bit ugly): https://github.com/tiiuae/ghaf/blob/main/modules/common/systemd/systemd-boot-double-dtb-buffer-size.patch Without the modification, because there is not any extra space, the UEFI will fail applying the overlay changes from its .dtbo files

Another way probably could be to modify the .dtb file to have some extra space in end of it