Kernel/aarch64: Perform an ISB after setting TTBR*_EL1

[spholz]

Changes to ARM system registers are not guaranteed to be visible until a context synchronization event, like performing an ISB.

[spholz]

@implicitfield does this also fix your page fault issue when running with hvf?

[implicitfield]

Yes, this does fix the page faults, and now the system appears to boot fine. Unfortunately, nothing appears on the framebuffer, and qemu itself crashes a few seconds after the system has finished starting up, as it hits this assertion. I found a somewhat similar issue on the qemu bug tracker, so could it be that we're not accessing some MMIO region properly somewhere?

Kernel log

[Kernel]: No early framebuffer console available, initializing dummy console [Kernel]: Starting SerenityOS... [Kernel]: DeviceTree: Attached device "timer" to driver ARMv8TimerDriver [Kernel]: DeviceTree: Attached device "intc@8000000" to driver GICDriver [Kernel]: TimeManagement: System timer: ARMv8 Timer 0.000 [Kernel]: KernelRng: Using bad entropy source TimeManagement 0.000 [#0 Kernel]: VFSRootContext(0): Root ("/") FileSystemID 1, Mounting RAMFS at inode 1:1 with flags 0 0.000 [#0 Kernel]: VFSRootContext(0): Context is artificially made, detach from global list 0.000 [#0 colonel(0:0)]: Scheduler[0]: idle loop running 0.237 [init_stage2(1:1)]: PCI [0000:00:00:00] PCI::HardwareID [1b36:0008] 0.237 [init_stage2(1:1)]: PCI [0000:00:01:00] PCI::HardwareID [8086:244e] 0.242 [init_stage2(1:1)]: PCI [0000:01:00:00] PCI::HardwareID [1b36:0010] 0.242 [init_stage2(1:1)]: PCI [0000:00:02:00] PCI::HardwareID [1234:1111] 0.310 [#0 init_stage2(1:1)]: Framebuffer Console: taking 3145728 bytes 0.314 [#0 init_stage2(1:1)]: Framebuffer Console: taking 3145728 bytes 0.314 [#0 init_stage2(1:1)]: VC 0: Resized to 113 x 48 0.314 [#0 init_stage2(1:1)]: VC 1: Resized to 113 x 48 0.314 [#0 init_stage2(1:1)]: VC 2: Resized to 113 x 48 0.314 [#0 init_stage2(1:1)]: VC 3: Resized to 113 x 48 0.314 [#0 init_stage2(1:1)]: VC 4: Resized to 113 x 48 0.314 [#0 init_stage2(1:1)]: VC 5: Resized to 113 x 48 0.320 [#0 init_stage2(1:1)]: AudioManagement: no audio controller was initialized. 0.324 [#0 VFS Sync Task(4:4)]: VFS SyncTask is running 2.322 [#0 init_stage2(1:1)]: GUIDPartitionTable: bad signature 0x00000000 0x00000000 2.322 [#0 init_stage2(1:1)]: VFSRootContext(1): Root ("/") FileSystemID 2, Mounting RAMFS at inode 2:1 with flags 0 2.322 [init_stage2(1:1)]: Ext2FS: Mount successful, setting superblock to error state. 2.326 [#0 init_stage2(1:1)]: VirtualFileSystem: Unmounting file system 2 for the last time... 2.326 [#0 init_stage2(1:1)]: VirtualFileSystem: Unmounting file system 2... 2.326 [#0 init_stage2(1:1)]: VFSRootContext(1): Root mount set to FileSystemID 3, Mounting Ext2FS at inode 3:2 with flags 0 2.326 [init_stage2(1:1)]: Write-protected kernel symbols after init. 2.326 [init_stage2(1:1)]: Unmapped 272 KiB of kernel text after init! :^) 2.326 [init_stage2(1:1)]: Running first user process: /init 2.326 [init_stage2(1:1)]: Init (first) process args: [ ] 2.342 [Network Task(7:7)]: NetworkTask: LoopbackAdapter network adapter found: hw={13:55:02:09:55:aa} 2.354 init(6): Spawning mount -a to mount all filesystems. 2.358 mount(8): Mounting all filesystems... 2.358 mount(8): Skipping mounting root 2.358 mount(8): Mounting /bin (bind) on /bin 2.358 [#0 mount(8:8)]: VFSRootContext(1): Bind-mounting inode 3:115272 at inode 3:115272 2.362 mount(8): Mounting /etc (bind) on /etc 2.362 [#0 mount(8:8)]: VFSRootContext(1): Bind-mounting inode 3:119379 at inode 3:119379 2.362 mount(8): Mounting /home (bind) on /home 2.362 [#0 mount(8:8)]: VFSRootContext(1): Bind-mounting inode 3:12290 at inode 3:12290 2.362 mount(8): Mounting /root (bind) on /root 2.362 [#0 mount(8:8)]: VFSRootContext(1): Bind-mounting inode 3:33364 at inode 3:33364 2.362 mount(8): Mounting /var (bind) on /var 2.362 [#0 mount(8:8)]: VFSRootContext(1): Bind-mounting inode 3:102996 at inode 3:102996 2.362 mount(8): Mounting /www (bind) on /www 2.362 [#0 mount(8:8)]: VFSRootContext(1): Bind-mounting inode 3:90708 at inode 3:90708 2.362 mount(8): Mounting /usr/Tests (bind) on /usr/Tests 2.362 [#0 mount(8:8)]: VFSRootContext(1): Bind-mounting inode 3:33092 at inode 3:33092 2.362 mount(8): Mounting /usr/local (bind) on /usr/local 2.362 [#0 mount(8:8)]: VFSRootContext(1): Bind-mounting inode 3:41304 at inode 3:41304 2.362 mount(8): Mounting /usr/Ports (bind) on /usr/Ports 2.362 [#0 mount(8:8)]: VFSRootContext(1): Bind-mounting inode 3:127325 at inode 3:127325 2.366 [#0 init(6:6)]: VirtualFileSystem: Remounting inode 3:2 2.366 [#0 init(6:6)]: VFSRootContext(1): FileSystemID 4, Mounting ProcFS at inode 3:8193 with flags 4 2.366 [#0 init(6:6)]: VFSRootContext(1): FileSystemID 5, Mounting SysFS at inode 3:86612 with flags 0 2.366 [#0 init(6:6)]: VFSRootContext(1): FileSystemID 6, Mounting RAMFS at inode 3:53844 with flags 262 2.366 [#0 init(6:6)]: VFSRootContext(1): FileSystemID 7, Mounting RAMFS at inode 3:119380 with flags 5 2.366 init(6): Creating /tmp/system/devicemap directory 2.366 [#0 init(6:6)]: VFSRootContext(1): FileSystemID 8, Mounting RAMFS at inode 7:3 with flags 7 2.374 [#0 init(6:6)]: VFSRootContext(1): FileSystemID 9, Mounting DevPtsFS at inode 6:10 with flags 0 2.374 [#0 init(6:6)]: VFSRootContext(1): FileSystemID 10, Mounting DevLoopFS at inode 6:11 with flags 0 2.378 init(6): Creating /tmp/coredump directory 2.378 init(6): Setting /tmp/coredump as the coredump directory 2.378 init(6): Creating /tmp/semaphore directory 2.395 SystemServer(6): Read system_mode: text 2.399 SystemServer(6): No such file or directory (errno=2) 2.399 SystemServer(6): No such file or directory (errno=2) 2.399 SystemServer(6): No such file or directory (errno=2) 2.399 SystemServer(6): Activating 4 services... 2.455 LookupServer(11): Using network config file at /etc/LookupServer.ini 2.476 NetworkServer(12): Interfaces to configure: [ ep1s0 ] 2.480 SystemServer(6): Service NetworkServer has exited with exit code 0 Assertion failed: (isv), function hvf_vcpu_exec, file hvf.c, line 1230.

[spholz]

Oh I also had a similar issue with KVM.
The way we copy the EDID data from the bochs VGA device caused a stage 2 (so guest paddr -> host paddr) translation fault with an invalid instruction syndrome (the instruction ldrb w3, [x1], #1 generated by the memcpy doesn't seem too complicated to me tho honestly).
This made KVM report an exit status of KVM_EXIT_ARM_NISV which QEMU translates to a normal page fault for the guest. So with KVM we get a normal page fault instead of an assertion failure in QEMU.

You also seem to be using the bochs VGA device but I don't think your problem is caused by the code copying the EDID, as init_stage2 seems to be finished for you.

Starting QEMU with -d unimp and -trace '*hvf*' and translating the reported pc value with addr2line -aCfipe Build/aarch64/Kernel/Kernel <pc value> could potentially help debugging this, assuming it's not caused by a memcpy.

[implicitfield]

So I tested running qemu with both of the options you mentioned, and I get the following additional (slightly corrupted) output before even the Starting SerenityOS message:

HVF errors

hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601030 pa=0x0000000009000030 isv=1 iswrite=1 s1ptw=0 len=4 srt=31] hvf_data_abort data abort: [pc=0x40000000 va=0x000000200260102c pa=0x000000000900002c isv=1 iswrite=1 s1ptw=0 len=4 srt=9] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601030 pa=0x0000000009000030 isv=1 iswrite=1 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] vf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] [hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] 3hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] 4hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ;hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] 1hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] mhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] [hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] Khvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ehvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] rhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] nhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ehvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] lhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ]hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] vf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] [hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] 0hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] mhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] :hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] Nhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ohvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ehvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ahvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] rhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] lhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] yhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] fhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] rhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ahvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] mhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ehvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] bhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] uhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] fhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] fhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ehvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] rhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] chvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ohvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] nhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] shvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ohvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] lhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ehvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ahvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] vhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ahvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ihvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] lhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ahvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] bhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] lhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ehvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ,hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ihvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] nhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ihvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] thvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ihvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ahvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] lhvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601018 pa=0x0000000009000018 isv=1 iswrite=0 s1ptw=0 len=4 srt=8] hvf_data_abort data abort: [pc=0x40000000 va=0x0000002002601000 pa=0x0000000009000000 isv=1 iswrite=1 s1ptw=0 len=4 srt=19] ihvf_data_abort data aborzing dummy console

pc always points to 0x40000000, so addr2line doesn't help here.

I did, however, find the location that crashes HVF with good old dbgln-debugging, and that is GenericFramebufferConsoleImpl's hide_cursor():

serenity/Kernel/Devices/GPU/Console/GenericFramebufferConsole.cpp

Lines 228 to 234 in 315ad73

	void GenericFramebufferConsoleImpl::hide_cursor()
	{
	auto offset_in_framebuffer = framebuffer_offset(m_x, m_y);
	offset_in_framebuffer.bytes += framebuffer_pitch() * (m_glyph_rows - 1);
	for (size_t glyph_column = 0; glyph_column < m_glyph_columns; glyph_column++)
	offset_in_framebuffer.pixels[glyph_column] = m_cursor_overriden_pixels[glyph_column];
	}

Since FramebufferOffset is a union, this function definitely invokes UB in C++ (it's a shame this doesn't trip UBSan though.)

Fixing that like this is enough to get system_mode=text fully working under HVF (although the framebuffer is awfully slow):

--- a/Kernel/Devices/GPU/Console/GenericFramebufferConsole.cpp
+++ b/Kernel/Devices/GPU/Console/GenericFramebufferConsole.cpp
@@ -228,9 +228,9 @@ GenericFramebufferConsoleImpl::FramebufferOffset GenericFramebufferConsoleImpl::
 void GenericFramebufferConsoleImpl::hide_cursor()
 {
     auto offset_in_framebuffer = framebuffer_offset(m_x, m_y);
-    offset_in_framebuffer.bytes += framebuffer_pitch() * (m_glyph_rows - 1);
+    auto offset = framebuffer_pitch() * (m_glyph_rows - 1) / sizeof(u32);
     for (size_t glyph_column = 0; glyph_column < m_glyph_columns; glyph_column++)
-        offset_in_framebuffer.pixels[glyph_column] = m_cursor_overriden_pixels[glyph_column];
+        offset_in_framebuffer.pixels[glyph_column + offset] = m_cursor_overriden_pixels[glyph_column];
 }
 
 void GenericFramebufferConsoleImpl::show_cursor()

[spholz]

Since FramebufferOffset is a union, this function definitely invokes UB in C++ (it's a shame this doesn't trip UBSan though.)

I think we are usually fine with using unions to do type punning and I wouldn't expect it to matter here either (I think GCC sees using unions for that as a language extension and Clang probably as well).
You probably just "tricked" clang into optimizing it less. Clang compiles this store to the framebuffer as a post-incrementing store: str w23, [x20], #4. GCC doesn't do that.
We will probably have to get both compilers to not emit loads/stores with writeback for MMIO regions somehow (we already had some discussions about that on discord but aren't completely sure how to solve this issue yet).
This problem is particularly annoying for MMIO framebuffers where for loops over the memory and memcpys are pretty common.

although the framebuffer is awfully slow

Can you try if setting enable_write_combine_optimization (the 3rd argument of the QEMUDisplayConnector base class constructor) to true helps?

(Those HVF errors aren't actual errors, that's just us writing to the serial console afaict)

[implicitfield]

I think we are usually fine with using unions to do type punning and I wouldn't expect it to matter here either (I think GCC sees using unions for that as a language extension and Clang probably as well).
You probably just "tricked" clang into optimizing it less. Clang compiles this store to the framebuffer as a post-incrementing store: str w23, [x20], #4. GCC doesn't do that.

Well, I guess it might be that as well, since it's easy to see how the compiler would optimize the current logic "better" than the modified version I posted above.

We will probably have to get both compilers to not emit loads/stores with writeback for MMIO regions somehow (we already had some discussions about that on discord but aren't completely sure how to solve this issue yet).
This problem is particularly annoying for MMIO framebuffers where for loops over the memory and memcpys are pretty common.

This does indeed seem pretty hard to fix given how deep this issue goes, since I just found that this issue can manifest all way in LibGfx::Painter::fill_rect() when that's used for drawing to WindowServer's backing bitmap.

Can you try if setting enable_write_combine_optimization (the 3rd argument of the QEMUDisplayConnector base class constructor) to true helps?

Enabling that didn't really seem to have any noticable impact on performance.

[spholz]

virtio-gpu might perform better and not have any of these problems since it doesn't use MMIO?

This does indeed seem pretty hard to fix given how deep this issue goes, since I just found that this issue can manifest all way in LibGfx::Painter::fill_rect() when that's used for drawing to WindowServer's backing bitmap.

I guess the only simple solution I can think of is double buffering MMIO framebuffers.

[implicitfield]

virtio-gpu might perform better and not have any of these problems since it doesn't use MMIO?

Well, I'm glad to report that performance with virtio-gpu is pretty much flawless, and that the whole system boots to desktop even when using HVF with this setup.