deploy: fd30083

ch10-00-swap-overview.html

@@ -356,24 +356,35 @@ <h4><a class="header" href="#flags-and-states" id="flags-and-states">Flags and S
 <p>Pages go from <code>Allocated</code> to <code>Swapped</code> based on the <code>swapper</code> observing that the kernel is low on memory, and calling a series of <code>EvictPage</code> calls to free up memory. It is always assumed that the kernel can allocate memory when necessary; as a last ditch the kernel can attempt to call <code>Trim</code> on the swapper, but this should only happen in extreme cases of memory pressure.</p>
 <p>Pages go from <code>Allocated</code> to <code>Reserved</code> when a process unmaps memory.</p>
 <p>When the <code>swapper</code> runs out of space, <code>WriteToSwap</code> panics with an OOM.</p>
+<h3><a class="header" href="#kernel-abi" id="kernel-abi">Kernel ABI</a></h3>
+<p>The swapper communicates with the kernel via two syscalls: <code>RegisterSwapper</code> and <code>SwapOp</code>. <code>RegisterSwapper</code> establishes the legitimacy of the swapper process; <code>SwapOp</code> is a wrapper around a swapper ABI that can change and evolve.</p>
 <h4><a class="header" href="#registerswapper-syscall" id="registerswapper-syscall">RegisterSwapper Syscall</a></h4>
 <p>The <code>swapper</code> registers with the kernel on a TOFU basis. The kernel reserves a single 128-bit <code>sid</code> with the target of the <code>swapper</code>, and it will trust the first process to use the <code>RegisterSwapper</code> syscall with its 128-bit random ID. Note that all the data necessary to setup the swapper is placed in the swapper's memory space by the loader, so the kernel does not need to marshall this.</p>
-<h4><a class="header" href="#evictpage-syscall" id="evictpage-syscall">EvictPage Syscall</a></h4>
-<p><code>EvictPage</code> is a syscall that only the <code>swapper</code> is allowed to call. It is a scalar <code>send</code> message, which contains the PID and address of the page to evict. Upon receipt, the kernel will:</p>
+<p>The arguments to <code>RegisterSwapper</code> are as follows:</p>
 <ul>
-<li>Change into the requested PID's address space</li>
-<li>Lookup the physical address of the evicted page</li>
-<li>Clear the <code>V</code> bit and set the <code>P</code> bit of the evicted page's PTE</li>
-<li>Mark the RPT entry as free</li>
-<li>Change into the swapper's address space</li>
-<li>Mutably lend the evicted physical page to the swapper with a <code>WriteToSwap</code> message</li>
-<li>Schedule the swapper to run</li>
+<li><code>s0</code>-<code>s3</code> are the 128-bit <code>sid</code></li>
+<li><code>handler</code> is the virtual address of the entry point for the blocking swap handler routine in the swapper's memory space</li>
+<li><code>state</code> is the virtual address of a pointer to the shared state between the swapper userspace and the swapper blocking handler</li>
 </ul>
-<h3><a class="header" href="#swapper-responsibilities" id="swapper-responsibilities">Swapper Responsibilities</a></h3>
-<p>It is the swapper's responsibility to maintain a structure that keeps track of every page in the <code>free RAM</code> pool. It builds this using the <code>AllocateAdvisory</code> messages.</p>
-<p>When the amount of free RAM falls below a certain threshold, the swapper will initiate a <code>Trim</code> operation. A kernel can also initiate a <code>Trim</code> as a last-ditch in case the <code>swapper</code> was starved of time and was unable to initiate a <code>Trim</code>, but this meant to be avoided.</p>
-<p>In a <code>Trim</code> operation, the swapper picks the pages it thinks will have the least performance impact and calls <code>EvictPage</code> to remove them. Initially, the swapper will have no way to know what pages are most important; it must use a heuristic to guess the first pages to evict. However, since it must track how frequently a page has been swapped with the <code>swap_count</code> field (necessary for encryption), it can rely upon this to build an eventual map of which pages to avoid.</p>
-<p>The swapper is also responsible for responding to <code>WriteToSwap</code> and <code>ReadFromSwap</code> calls.</p>
+<h4><a class="header" href="#swapop-syscall" id="swapop-syscall">SwapOp Syscall</a></h4>
+<p>The <code>SwapOp</code> syscall that encodes a private ABI between the swapper and the kernel. The idea is that this ABI can rapidly evolve without having to update the syscall table, which would require an update to the Xous package itself. The <code>SwapOp</code> syscall has arguments consisting of the op itself, which is coded as the numerical representation of <code>SwapAbi</code> (below), and up to 6 generic <code>usize</code> arguments that have a meaning depending on the <code>SwapAbi</code> code.</p>
+<p>The <code>SwapAbi</code> may change rapidly, so please refer to the code for the latest details, but below gives you an idea of what is inside the ABI.</p>
+<pre><pre class="playground"><code class="language-rust">
+<span class="boring">#![allow(unused)]
+</span><span class="boring">fn main() {
+</span>pub enum SwapAbi {
+    Invalid = 0,
+    Evict = 1,
+    GetFreePages = 2,
+    FetchAllocs = 3,
+    StealPage = 5,
+    ReleaseMemory = 6,
+}
+<span class="boring">}
+</span></code></pre></pre>
+<p>There are two basic modes of operation supported by the swapper. One is a userspace-driven <code>Evict</code>ion of pages, and the other is a blocking handler driven <code>Steal</code>/<code>Release</code> cycle. The <code>Evict</code> mode is a soft-OOM handler, nicknamed the <code>OOM Doom</code> handler, where a userspace program tries to free up memory using all the tools available in rust <code>std</code> (including more heap allocations!) without blocking forward progress (i.e., it is pre-emptable). It can be more deliberative and analytical about which pages to remove, and it invokes the kernel with an <code>Evict</code> call which will atomically remove one page at a time, stealing the memory from the process, writing it to swap (by doing a re-entrant call back into the swapper's userspace blocking handler), and then releasing the memory.</p>
+<p>The <code>Steal</code>/<code>Release</code> mode is invoked on a hard-OOM, i.e. when we literally have 0 free pages of memory left in the system. This handler is significantly more constrained on what it can do, and it operates in a fully blocking context. In this case, the userspace tries to aggressively swap memory out by <code>Steal</code>ing pages from other processes, writing their pages to swap, and then <code>Release</code>ing their memory from the physical memory allocation table. It will fairly indiscriminately steal memory from processes until enough memory is free to allow forward progress on other operations.</p>
+<p>Most of the interesting code for the swapper is split between the kernel stub (inside kernel/src/swap.rs) and the userspace service (services/xous-swapper). The userspace service itself is split into the blocking handler and the regular preemptable handler. There are also a couple of modifications to the architecture-specific implementations (kernel/src/arch/riscv/irq.rs and kernel/src/arch/riscv/mem.rs) to shim the swap into core memory routines. Keep in mind at least half the effort for swap is in the loader, which is responsible for setting up and initializing all the relevant data structures so that swap is even possible.</p>
 
                     </main>
 

print.html

@@ -3926,24 +3926,35 @@ <h4><a class="header" href="#flags-and-states" id="flags-and-states">Flags and S
 <p>Pages go from <code>Allocated</code> to <code>Swapped</code> based on the <code>swapper</code> observing that the kernel is low on memory, and calling a series of <code>EvictPage</code> calls to free up memory. It is always assumed that the kernel can allocate memory when necessary; as a last ditch the kernel can attempt to call <code>Trim</code> on the swapper, but this should only happen in extreme cases of memory pressure.</p>
 <p>Pages go from <code>Allocated</code> to <code>Reserved</code> when a process unmaps memory.</p>
 <p>When the <code>swapper</code> runs out of space, <code>WriteToSwap</code> panics with an OOM.</p>
+<h3><a class="header" href="#kernel-abi" id="kernel-abi">Kernel ABI</a></h3>
+<p>The swapper communicates with the kernel via two syscalls: <code>RegisterSwapper</code> and <code>SwapOp</code>. <code>RegisterSwapper</code> establishes the legitimacy of the swapper process; <code>SwapOp</code> is a wrapper around a swapper ABI that can change and evolve.</p>
 <h4><a class="header" href="#registerswapper-syscall" id="registerswapper-syscall">RegisterSwapper Syscall</a></h4>
 <p>The <code>swapper</code> registers with the kernel on a TOFU basis. The kernel reserves a single 128-bit <code>sid</code> with the target of the <code>swapper</code>, and it will trust the first process to use the <code>RegisterSwapper</code> syscall with its 128-bit random ID. Note that all the data necessary to setup the swapper is placed in the swapper's memory space by the loader, so the kernel does not need to marshall this.</p>
-<h4><a class="header" href="#evictpage-syscall" id="evictpage-syscall">EvictPage Syscall</a></h4>
-<p><code>EvictPage</code> is a syscall that only the <code>swapper</code> is allowed to call. It is a scalar <code>send</code> message, which contains the PID and address of the page to evict. Upon receipt, the kernel will:</p>
-<ul>
-<li>Change into the requested PID's address space</li>
-<li>Lookup the physical address of the evicted page</li>
-<li>Clear the <code>V</code> bit and set the <code>P</code> bit of the evicted page's PTE</li>
-<li>Mark the RPT entry as free</li>
-<li>Change into the swapper's address space</li>
-<li>Mutably lend the evicted physical page to the swapper with a <code>WriteToSwap</code> message</li>
-<li>Schedule the swapper to run</li>
-</ul>
-<h3><a class="header" href="#swapper-responsibilities" id="swapper-responsibilities">Swapper Responsibilities</a></h3>
-<p>It is the swapper's responsibility to maintain a structure that keeps track of every page in the <code>free RAM</code> pool. It builds this using the <code>AllocateAdvisory</code> messages.</p>
-<p>When the amount of free RAM falls below a certain threshold, the swapper will initiate a <code>Trim</code> operation. A kernel can also initiate a <code>Trim</code> as a last-ditch in case the <code>swapper</code> was starved of time and was unable to initiate a <code>Trim</code>, but this meant to be avoided.</p>
-<p>In a <code>Trim</code> operation, the swapper picks the pages it thinks will have the least performance impact and calls <code>EvictPage</code> to remove them. Initially, the swapper will have no way to know what pages are most important; it must use a heuristic to guess the first pages to evict. However, since it must track how frequently a page has been swapped with the <code>swap_count</code> field (necessary for encryption), it can rely upon this to build an eventual map of which pages to avoid.</p>
-<p>The swapper is also responsible for responding to <code>WriteToSwap</code> and <code>ReadFromSwap</code> calls.</p>
+<p>The arguments to <code>RegisterSwapper</code> are as follows:</p>
+<ul>
+<li><code>s0</code>-<code>s3</code> are the 128-bit <code>sid</code></li>
+<li><code>handler</code> is the virtual address of the entry point for the blocking swap handler routine in the swapper's memory space</li>
+<li><code>state</code> is the virtual address of a pointer to the shared state between the swapper userspace and the swapper blocking handler</li>
+</ul>
+<h4><a class="header" href="#swapop-syscall" id="swapop-syscall">SwapOp Syscall</a></h4>
+<p>The <code>SwapOp</code> syscall that encodes a private ABI between the swapper and the kernel. The idea is that this ABI can rapidly evolve without having to update the syscall table, which would require an update to the Xous package itself. The <code>SwapOp</code> syscall has arguments consisting of the op itself, which is coded as the numerical representation of <code>SwapAbi</code> (below), and up to 6 generic <code>usize</code> arguments that have a meaning depending on the <code>SwapAbi</code> code.</p>
+<p>The <code>SwapAbi</code> may change rapidly, so please refer to the code for the latest details, but below gives you an idea of what is inside the ABI.</p>
+<pre><pre class="playground"><code class="language-rust">
+<span class="boring">#![allow(unused)]
+</span><span class="boring">fn main() {
+</span>pub enum SwapAbi {
+    Invalid = 0,
+    Evict = 1,
+    GetFreePages = 2,
+    FetchAllocs = 3,
+    StealPage = 5,
+    ReleaseMemory = 6,
+}
+<span class="boring">}
+</span></code></pre></pre>
+<p>There are two basic modes of operation supported by the swapper. One is a userspace-driven <code>Evict</code>ion of pages, and the other is a blocking handler driven <code>Steal</code>/<code>Release</code> cycle. The <code>Evict</code> mode is a soft-OOM handler, nicknamed the <code>OOM Doom</code> handler, where a userspace program tries to free up memory using all the tools available in rust <code>std</code> (including more heap allocations!) without blocking forward progress (i.e., it is pre-emptable). It can be more deliberative and analytical about which pages to remove, and it invokes the kernel with an <code>Evict</code> call which will atomically remove one page at a time, stealing the memory from the process, writing it to swap (by doing a re-entrant call back into the swapper's userspace blocking handler), and then releasing the memory.</p>
+<p>The <code>Steal</code>/<code>Release</code> mode is invoked on a hard-OOM, i.e. when we literally have 0 free pages of memory left in the system. This handler is significantly more constrained on what it can do, and it operates in a fully blocking context. In this case, the userspace tries to aggressively swap memory out by <code>Steal</code>ing pages from other processes, writing their pages to swap, and then <code>Release</code>ing their memory from the physical memory allocation table. It will fairly indiscriminately steal memory from processes until enough memory is free to allow forward progress on other operations.</p>
+<p>Most of the interesting code for the swapper is split between the kernel stub (inside kernel/src/swap.rs) and the userspace service (services/xous-swapper). The userspace service itself is split into the blocking handler and the regular preemptable handler. There are also a couple of modifications to the architecture-specific implementations (kernel/src/arch/riscv/irq.rs and kernel/src/arch/riscv/mem.rs) to shim the swap into core memory routines. Keep in mind at least half the effort for swap is in the loader, which is responsible for setting up and initializing all the relevant data structures so that swap is even possible.</p>
 
                     </main>