Testing (catching memory-safety bugs) with Address Sanitizer

[dimakuv]

Build:

export CC="clang"
export CXX="clang++"
export AS="clang"

# see https://github.com/gramineproject/gramine/blob/master/.ci/lib/config-clang.jenkinsfile
export ARCH_LIBDIR="/lib/x86_64-linux-gnu"

rm -rf buil*-release
rm -rf buil*-debug

meson setup build-debug/ --buildtype=debug --werror -Dskeleton=enabled \
  -Ddirect=enabled -Dsgx=enabled -Dvm=enabled -Dtdx=enabled \
  -Dasan=enabled -Dtests=enabled --prefix=$PWD/built-debug

ninja -C build-debug/ install

NOTE: I needed at least Clang 12. Clang 10 compiled but incorrectly built the BIOS, which crashed QEMU.

[dimakuv]

Explanations on the implementation

We already have ASAN enabled inside Gramine-SGX, and it can be easily adapted to Gramine-TDX.

Here are the relevant ASAN files:

• https://github.com/gramineproject/gramine/blob/master/common/include/asan.h
• https://github.com/gramineproject/gramine/blob/master/common/src/asan.c

We don't want to modify any Gramine constants for ASAN. This means we are bound to use the [1.5TB, 1.5TB + 1<<45) memory region for ASAN shadow memory.

Clearly, our 1:1 virtual-to-physical mapping does not work well in this case. So we need to have a separate set of page tables that will cover (a reasonable part) of this shadow memory region.

Recall that ASAN's real-to-shadow memory mapping is like this:

asan_mem_to_shadow(const void *addr) {
    shadow_addr = addr >> ASAN_SHADOW_SHIFT + ASAN_SHADOW_START;
}

If we assume that normal Gramine VMs will use up to 32GB of memory, then we only need a part of the shadow memory region that covers the top-most address, which is 32GB. By putting real values in the formula, we get:

    last_shadow_addr = ((32*1024*1024*1024) >> 3) + 1.5 * 1024*1024*1024*1024 = 1653562408960

So we have:

• first_shadow_addr = 1649267441664.0 (1.5TB)
• last_shadow_addr = 1653562408960 (slightly more than 1.5TB)

This gives us the size of the shadow memory region that we actually need (to cover first 32GB of app's + Gramine's really used memory):

   used_shadow_size = last_shadow_addr - first_shadow_addr = 4294967296

This is 4GB. Which is no surprise because to cover 32GB, ASAN requires 1/8 of it, which is 4GB.

How much memory space we need for page tables to cover 4GB of this actually-used shadow memory region? Since each 4KB-sized page in PT has 512 PTEs, and each PTE describes a 4KB of memory, this means that PTs ratio is 512:1. Thus:

    required_page_tables_size = used_shadow_size / 512 = 8388608.0

This is 8MB. So in the end, to cover 32GB app memory with 4GB ASAN shadow memory, we need 8MB of page tables that must be correctly filled up (present, RW permissions, covering [1.5TB, 1.5TB + 4GB)) at Gramine boot time.

Gramine VM logic previously used 136MB of page tables, used to cover up to 64GB of memory. Technically, we need only 129MB to cover 63GB, but we added a bit more slack with additional 7MB here.

However, 136MB of page tables was not enough if we want to have a 64GB addr space and also shadow addr space. So we need to increase the size of page tables to smth bigger. This should be trivial because adjacent to page tables, we have virtio queues shared memory.

Virtio queues region was 248MB, which is way too much for our current usages of them (they maybe occupy up to 64MB, no more than that). So we can easily resize these two regions, increasing page tables by another e.g. 8MB and decreasing the virtio queues region.

---

Since we already have a limit of 64GB on possible address space, it makes sense to allow the same limit on ASAN shadow memory. So ASAN shadow memory will need 8GB, and we'll require 16MB of page tables .

So in general, we want to have these characteristics:

1. 1:1 normal page tables that cover the address space of apps + Gramine -- [0, 64GB).
2. 1:1-with-offset "ASAN shadow memory" page tables that cover the shadow address space of ASAN -- [1.5TB, 1.5TB + 8GB).

The former requires around 129MB, and the latter requires around 17MB. So a total of 146MB for page tables should be enough.

[dimakuv]

Here's a diff of how to test ASan manually:

diff --git a/pal/src/host/vm-common/kernel_memory.c b/pal/src/host/vm-common/kernel_memory.c
index ebfb72dd..88209332 100644
--- a/pal/src/host/vm-common/kernel_memory.c
+++ b/pal/src/host/vm-common/kernel_memory.c
@@ -26,6 +26,33 @@
 #include "kernel_multicore.h"
 #include "kernel_virtio.h"

+/* ------------------- */
+__attribute__((no_sanitize("undefined")))
+static int run_test_asan_heap(void) {
+    char* buf = malloc(30);
+    char* volatile c = buf + 30;
+    *c = 1;
+    free(buf);
+    return 0;
+}
+
+__attribute__((no_sanitize("undefined")))
+static int run_test_asan_stack(void) {
+    char buf[30];
+    char* volatile c = buf + 30;
+    *c = 1;
+    return 0;
+}
+
+__attribute__((no_sanitize("undefined")))
+static int run_test_asan_global(void) {
+    static char buf[30];
+    char* volatile c = buf + 30;
+    *c = 1;
+    return 0;
+}
+/* ------------------- */
+
 static_assert(PAGE_SIZE == 4096, "unexpected PAGE_SIZE (expected 4K)");

 /* Beginning of the page table hierarchy */
@@ -550,6 +577,19 @@ int memory_tighten_permissions(void) {
             return -PAL_ERROR_DENIED;
     }

+
+       debug_serial_io_write("--- before testing Address Sanitizer");
+#if 1
+       (void)run_test_asan_heap();
+#endif
+#if 0
+    (void)run_test_asan_stack();
+#endif
+#if 0
+       (void)run_test_asan_global();
+#endif
+       debug_serial_io_write("--- after testing Address Sanitizer");
+
     return 0;
 }