Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

1.9.1 does not detect NVME disks anymore (Intel N5000) #10070

Open
runningman84 opened this issue Dec 29, 2024 · 0 comments
Open

1.9.1 does not detect NVME disks anymore (Intel N5000) #10070

runningman84 opened this issue Dec 29, 2024 · 0 comments

Comments

@runningman84
Copy link

runningman84 commented Dec 29, 2024 •
edited
Loading

Bug Report

I upgraded my node from 1.8.4 to 1.9.1 it boots fine but does not find the nvme device anymore:

➜  k8s-prod git:(main) ✗ talosctl get disks -n 10.0.20.81 -i
NODE   NAMESPACE   TYPE   ID      VERSION   SIZE     READ ONLY   TRANSPORT   ROTATIONAL   WWID   MODEL   SERIAL
       runtime     Disk   loop0   1         164 kB   true                                                
       runtime     Disk   loop1   1         4.1 kB   true                                                
       runtime     Disk   loop2   1         7.5 MB   true                                                
       runtime     Disk   loop3   1         57 kB    true                                                
       runtime     Disk   loop4   1         2.7 MB   true                                                
       runtime     Disk   loop5   1         2.1 MB   true                                                
       runtime     Disk   loop6   1         4.1 kB   true                                                
       runtime     Disk   loop7   1         127 kB   true                                                
       runtime     Disk   loop8   1         74 MB    true

The same command on another node with the same hardware still running 1.8.4 shows that:

➜  k8s-prod git:(main) ✗ talosctl get disks -n 10.0.20.82   
WARNING: 10.0.20.82: server version 1.8.3 is older than client version 1.9.1
NODE         NAMESPACE   TYPE   ID        VERSION   SIZE     READ ONLY   TRANSPORT   ROTATIONAL   WWID                                   MODEL                   SERIAL
10.0.20.82   runtime     Disk   dm-0      1         88 MB    false                                                                                               
10.0.20.82   runtime     Disk   dm-1      1         499 GB   false                                                                                               
10.0.20.82   runtime     Disk   loop0     1         156 kB   true                                                                                                
10.0.20.82   runtime     Disk   loop1     1         4.1 kB   true                                                                                                
10.0.20.82   runtime     Disk   loop2     1         6.7 MB   true                                                                                                
10.0.20.82   runtime     Disk   loop3     1         57 kB    true                                                                                                
10.0.20.82   runtime     Disk   loop4     1         2.6 MB   true                                                                                                
10.0.20.82   runtime     Disk   loop5     1         4.1 kB   true                                                                                                
10.0.20.82   runtime     Disk   loop6     1         4.1 kB   true                                                                                                
10.0.20.82   runtime     Disk   loop7     1         119 kB   true                                                                                                
10.0.20.82   runtime     Disk   loop8     1         75 MB    true                                                                                                
10.0.20.82   runtime     Disk   nvme0n1   1         500 GB   false       nvme                     eui.002538dc31a109cd                   Samsung SSD 980 500GB   XXXXXXXXXXXXXXXX

These are my extensions:

          officialExtensions:
            - siderolabs/gasket-driver
            - siderolabs/intel-ucode
            - siderolabs/i915 (-ucode in 1.8.x)
            - siderolabs/iscsi-tools
            - siderolabs/util-linux-tools
            - siderolabs/zfs

Description

Unfortunatly most of the other commands like dmesg or support do not work in maintenance mode. What else can I provide? I could provide you with the support bundle if I reboot the 10.0.20.81 node with the old 1.8.4 version...

Logs

This is the lspci output running talos 1.8.4:

root@master1:~# lspci  -v
00:00.0 Host bridge: Intel Corporation Gemini Lake Host Bridge (rev 03)
        Flags: bus master, fast devsel, latency 0
lspci: Unable to load libkmod resources: error -2

00:00.3 System peripheral: Intel Corporation Celeron/Pentium Silver Processor Gaussian Mixture Model (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: fast devsel, IRQ 23
        Memory at a1530000 (64-bit, non-prefetchable) [disabled] [size=4K]
        Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
        Capabilities: [a0] Vendor Specific Information: Len=14 <?>
        Capabilities: [dc] Power Management version 2
        Capabilities: [f0] PCI Advanced Features

00:02.0 VGA compatible controller: Intel Corporation GeminiLake [UHD Graphics 605] (rev 03) (prog-if 00 [VGA controller])
        DeviceName:  Onboard IGD
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 144
        Memory at a0000000 (64-bit, non-prefetchable) [size=16M]
        Memory at 90000000 (64-bit, prefetchable) [size=256M]
        I/O ports at f000 [size=64]
        Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [d0] Power Management version 2
        Capabilities: [100] Process Address Space ID (PASID)
        Capabilities: [200] Address Translation Service (ATS)
        Capabilities: [300] Page Request Interface (PRI)
        Kernel driver in use: i915

00:0e.0 Audio device: Intel Corporation Celeron/Pentium Silver Processor High Definition Audio (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 25
        Memory at a1510000 (64-bit, non-prefetchable) [size=16K]
        Memory at a1000000 (64-bit, non-prefetchable) [size=1M]
        Capabilities: [50] Power Management version 3
        Capabilities: [80] Vendor Specific Information: Len=14 <?>
        Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00

00:0f.0 Communication controller: Intel Corporation Celeron/Pentium Silver Processor Trusted Execution Engine Interface (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 255
        Memory at a152f000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [50] Power Management version 3
        Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [a4] Vendor Specific Information: Len=14 <?>

00:12.0 SATA controller: Intel Corporation Celeron/Pentium Silver Processor SATA Controller (rev 03) (prog-if 01 [AHCI 1.0])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 125
        Memory at a1514000 (32-bit, non-prefetchable) [size=8K]
        Memory at a152e000 (32-bit, non-prefetchable) [size=256]
        I/O ports at f090 [size=8]
        I/O ports at f080 [size=4]
        I/O ports at f060 [size=32]
        Memory at a152d000 (32-bit, non-prefetchable) [size=2K]
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [70] Power Management version 3
        Capabilities: [a8] SATA HBA v1.0
        Kernel driver in use: ahci

00:13.0 PCI bridge: Intel Corporation Gemini Lake PCI Express Root Port (rev f3) (prog-if 00 [Normal decode])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 120
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
        I/O behind bridge: [disabled] [16-bit]
        Memory behind bridge: a1400000-a14fffff [size=1M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Capabilities: [40] Express Root Port (Slot+), MSI 00
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [90] Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Capabilities: [a0] Power Management version 3
        Capabilities: [100] Null
        Capabilities: [140] Access Control Services
        Capabilities: [150] Null
        Capabilities: [200] Null
        Kernel driver in use: pcieport

00:13.2 PCI bridge: Intel Corporation Gemini Lake PCI Express Root Port (rev f3) (prog-if 00 [Normal decode])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 121
        Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
        I/O behind bridge: [disabled] [16-bit]
        Memory behind bridge: a1100000-a12fffff [size=2M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Capabilities: [40] Express Root Port (Slot+), MSI 00
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [90] Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Capabilities: [a0] Power Management version 3
        Capabilities: [100] Null
        Capabilities: [140] Access Control Services
        Capabilities: [150] Null
        Capabilities: [200] Null
        Kernel driver in use: pcieport

00:13.3 PCI bridge: Intel Corporation Gemini Lake PCI Express Root Port (rev f3) (prog-if 00 [Normal decode])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 122
        Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
        I/O behind bridge: e000-efff [size=4K] [16-bit]
        Memory behind bridge: a1300000-a13fffff [size=1M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Capabilities: [40] Express Root Port (Slot+), MSI 00
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [90] Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Capabilities: [a0] Power Management version 3
        Capabilities: [100] Null
        Capabilities: [140] Access Control Services
        Capabilities: [150] Null
        Capabilities: [200] Null
        Kernel driver in use: pcieport

00:15.0 USB controller: Intel Corporation Celeron/Pentium Silver Processor USB 3.0 xHCI Controller (rev 03) (prog-if 30 [XHCI])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: medium devsel, IRQ 123
        Memory at a1500000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [70] Power Management version 2
        Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
        Capabilities: [90] Vendor Specific Information: Len=14 <?>
        Kernel driver in use: xhci_hcd

00:16.0 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 0 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 27
        Memory at a152c000 (64-bit, non-prefetchable) [size=4K]
        Memory at a152b000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:16.1 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 1 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 28
        Memory at a152a000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1529000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:16.2 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 2 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 29
        Memory at a1528000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1527000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:16.3 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 3 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 30
        Memory at a1526000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1525000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:17.0 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 4 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 31
        Memory at a1524000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1523000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:17.1 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 5 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 32
        Memory at a1522000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1521000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:17.2 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 6 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 33
        Memory at a1520000 (64-bit, non-prefetchable) [size=4K]
        Memory at a151f000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:17.3 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 7 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 34
        Memory at a151e000 (64-bit, non-prefetchable) [size=4K]
        Memory at a151d000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:19.0 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor Serial IO SPI Host Controller (rev 03)
        DeviceName:  Onboard LAN
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 35
        Memory at a151c000 (64-bit, non-prefetchable) [size=4K]
        Memory at a151b000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:19.1 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor Serial IO SPI Host Controller (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 36
        Memory at a151a000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1519000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:19.2 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor Serial IO SPI Host Controller (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 37
        Memory at a1518000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1517000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:1f.0 ISA bridge: Intel Corporation Celeron/Pentium Silver Processor LPC Controller (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, medium devsel, latency 0

00:1f.1 SMBus: Intel Corporation Celeron/Pentium Silver Processor Gaussian Mixture Model (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: medium devsel, IRQ 20
        Memory at a1516000 (64-bit, non-prefetchable) [size=256]
        I/O ports at f040 [size=32]
        Kernel driver in use: i801_smbus

03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 980 (prog-if 02 [NVM Express])
        Subsystem: Samsung Electronics Co Ltd Device a801
        Flags: bus master, fast devsel, latency 0, IRQ 22
        Memory at a1400000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [b0] MSI-X: Enable+ Count=13 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [158] Power Budgeting <?>
        Capabilities: [168] Secondary PCI Express
        Capabilities: [188] Latency Tolerance Reporting
        Capabilities: [190] L1 PM Substates
        Kernel driver in use: nvme

05:00.0 System peripheral: Global Unichip Corp. Coral Edge TPU (prog-if ff)
        Subsystem: Global Unichip Corp. Coral Edge TPU
        Flags: bus master, fast devsel, latency 0, IRQ 20
        Memory at a1200000 (64-bit, prefetchable) [size=16K]
        Memory at a1100000 (64-bit, prefetchable) [size=1M]
        Capabilities: [80] Express Endpoint, MSI 00
        Capabilities: [d0] MSI-X: Enable+ Count=128 Masked-
        Capabilities: [e0] MSI: Enable- Count=1/32 Maskable- 64bit+
        Capabilities: [f8] Power Management version 3
        Capabilities: [100] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
        Capabilities: [108] Latency Tolerance Reporting
        Capabilities: [110] L1 PM Substates
        Capabilities: [200] Advanced Error Reporting
        Kernel driver in use: apex

06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 21
        I/O ports at e000 [size=256]
        Memory at a1304000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1300000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
        Capabilities: [170] Latency Tolerance Reporting
        Capabilities: [178] L1 PM Substates
        Kernel driver in use: r8169

Environment

  • Talos version: [talosctl version --nodes <problematic nodes>] 1.9.1
  • Kubernetes version: [kubectl version --short] 1.30.8
  • Platform: bare metal
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@runningman84