Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

remote thread creation needs to fail more gently #102

Open
bxatnarf opened this issue Mar 30, 2020 · 1 comment
Open

remote thread creation needs to fail more gently #102

bxatnarf opened this issue Mar 30, 2020 · 1 comment

Comments

@bxatnarf
Copy link
Collaborator

bxatnarf commented Mar 30, 2020 •
edited
Loading

When the following is invoked from host 1, execution on both hosts hang, although both kernels are still responsive. Host 0 (remote host) complains of a recursive fault. Then when ps -aux is invoked on host 1, the host 1 kernel panic occurs.
The failure is because popcorn does not support remote thread creation. It would be best to catch remote thread creation sooner and fail more gently.

Triggering code:

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include "migrate.h"

#define NUM_THREADS 2

int data[NUM_THREADS];

void *do_it(void *t)
{
  int ret = 0;
  long long i = (long long)t;
  data[i] = 10;
  pthread_exit (&ret);
}

int
main (int argc, char *argv[])
{
  long long i;
  pthread_t threads[NUM_THREADS];

  migrate(0, NULL, NULL);

  for (i = 0; i < NUM_THREADS; i++)
    {
      pthread_create(&threads[i], NULL, do_it, (void *)i);
    }

  for (i = 0; i < NUM_THREADS; i++)
    {
      int status = 0;
      pthread_join (threads[i], (void **) &status);
    }

  migrate(1, NULL, NULL);

  printf ("%d\n", data[0]);
  return data[0];
}

Log on host 0

[  T640] remote_worker_main: [640] for [639/1]
[  T640] remote_worker_main: [640] /home/popcorn/github-100
[  T641] remote_thread_main [641] started for [639/1]
[  T641] restore_thread_info [641] ip 401efd
[  T641] restore_thread_info [641] sp 7fffffffe2d0 bp 408290
[  T641] 
[  T641] ####### MIGRATED - [641/0] from [639/1]
[  T641] 
[  T641] ## VMAFAULT [641] 401efd 401efd
[  T641]   [641] 401000 ->[639/1]
[  T641]   [641] 0 401000
[  T641]   [641] /home/popcorn/github-100 + 1
[  T641]   [641] map+ 401000 499000
[  T641]   [641] map0 401000 -- 499000 @ 1, 5
[  T641] 
[  T641] ## PAGEFAULT [641] 401000 R 401efd 154 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## PAGEFAULT [641] 401000 R 401efd 170 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## VMAFAULT [641] 7fffffffe308 401f00
[  T641]   [641] 7fffffffe000 ->[639/1]
[  T641]   [641] 0 7fffffffe000
[  T641]   [641] map+ 7ffffffde000 7ffffffff000
[  T641]   [641] map0 7ffffffde000 -- 7ffffffff000 @ 7ffffffde, 3
[  T641] 
[  T641] ## PAGEFAULT [641] 7fffffffe000 R 401f00 54 0
[  T641]  =[641] 7fffffffe000 00000000350a4be7
[  T641]   [641] ->[639/1] 7fffffffe000 401f00
[  T639]   [641] <-[639/1] 7fffffffe000 0
[  T641] >>[641] 7fffffffe000 00000000350a4be7
[  T641] 
[  T641] ## VMAFAULT [641] 4ce8a8 401fd7
[  T641]   [641] 4ce000 ->[639/1]
[  T641]   [641] 7ffffffde000 4ce000
[  T641]   [641] map+ 4c8000 4f1000
[  T641]   [641] map0 4c8000 -- 4f1000 @ 4c8, 3
[  T641] 
[  T641] ## PAGEFAULT [641] 4ce000 R 401fd7 54 0
[  T641]  =[641] 4ce000 00000000350a4be7
[  T641]   [641] ->[639/1] 4ce000 401fd7
[  T639]   [641] <-[639/1] 4ce000 0
[  T641] >>[641] 4ce000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 7fffffffe000 W 401cbd 55 8000000000000025
[  T641]  =[641] 7fffffffe000 00000000350a4be7
[  T641]   [641] ->[639/1] 7fffffffe000 401cbd
[  T639]   [641] <-[639/1] 7fffffffe000 4000
[  T641] >>[641] 7fffffffe000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 4cb000 W 4035fc 55 0
[  T641]  =[641] 4cb000 00000000350a4be7
[  T641]   [641] ->[639/1] 4cb000 4035fc
[  T639]   [641] <-[639/1] 4cb000 0
[  T641] >>[641] 4cb000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 4cc000 R 40360a 54 0
[  T641]  =[641] 4cc000 00000000350a4be7
[  T641]   [641] ->[639/1] 4cc000 40360a
[  T639]   [641] <-[639/1] 4cc000 0
[  T641] >>[641] 4cc000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 44c000 R 44c920 154 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## PAGEFAULT [641] 44c000 R 44c920 170 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## VMAFAULT [641] 4c6198 44c924
[  T641]   [641] 4c6000 ->[639/1]
[  T641]   [641] 4c8000 4c6000
[  T641]   [641] /home/popcorn/github-100 + c1
[  T641]   [641] map+ 4c2000 4c8000
[  T641]   [641] map0 4c2000 -- 4c8000 @ c1, 3
[  T641] 
[  T641] ## PAGEFAULT [641] 4c6000 R 44c924 54 0
[  T641]  =[641] 4c6000 0000000025699d3b
[  T641]   [641] ->[639/1] 4c6000 44c924
[  T639]   [641] <-[639/1] 4c6000 0
[  T641] >>[641] 4c6000 0000000025699d3b
[  T641] 
[  T641] ## PAGEFAULT [641] 4c7000 W 40376a 55 0
[  T641]  =[641] 4c7000 0000000025699d3b
[  T641]   [641] ->[639/1] 4c7000 40376a
[  T639]   [641] <-[639/1] 4c7000 0
[  T641] >>[641] 4c7000 0000000025699d3b
[  T641] 
[  T641] ## PAGEFAULT [641] 4c5000 R 40377c 54 0
[  T641]  =[641] 4c5000 0000000025699d3b
[  T641]   [641] ->[639/1] 4c5000 40377c
[  T639]   [641] <-[639/1] 4c5000 0
[  T641] >>[641] 4c5000 0000000025699d3b
[  T641] 
[  T641] ## VMA mmap [641] 0 - 801000, 0 20022
[  T641]   [641] 0 7ffff77fa000 -- 7ffff7ffb000
[  T641]   [641] map+ 7ffff77fa000 7ffff7ffb000
[  T641]   [641] map0 7ffff77fa000 -- 7ffff7ffb000 @ 0, 0
[  T641] 
[  T641] VMA mprotect [641] 7ffff77fb000 800000 3
[  T641]   [641] 0 7ffff77fb000 -- 7ffff7ffb000 3
[  T641] 
[  T641] ## PAGEFAULT [641] 7ffff7ffa000 W 403f53 55 0
[  T641]  =[641] 7ffff7ffa000 0000000025699d3b
[  T641]   [641] ->[639/1] 7ffff7ffa000 403f53
[  T639]   [641] <-[639/1] 7ffff7ffa000 0
[  T641] >>[641] 7ffff7ffa000 0000000025699d3b
[  T641] 
[  T641] ## PAGEFAULT [641] 4cd000 W 403f8f 55 0
[  T641]  =[641] 4cd000 0000000025699d3b
[  T641]   [641] ->[639/1] 4cd000 403f8f
[  T639]   [641] <-[639/1] 4cd000 0
[  T641] >>[641] 4cd000 0000000025699d3b
[  T641] 
[  T641] ## PAGEFAULT [641] 4cc000 W 403fa0 55 8000000000000025
[  T641]  =[641] 4cc000 0000000025699d3b
[  T641]   [641] ->[639/1] 4cc000 403fa0
[  T639]   [641] <-[639/1] 4cc000 4000
[  T641] >>[641] 4cc000 0000000025699d3b
[  T641] 
[  T641] ## PAGEFAULT [641] 422000 R 422e50 154 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## PAGEFAULT [641] 422000 R 422e50 170 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## PAGEFAULT [641] 4cf000 R 422ebd 54 0
[  T641]  =[641] 4cf000 00000000350a4be7
[  T641]   [641] ->[639/1] 4cf000 422ebd
[  T639]   [641] <-[639/1] 4cf000 0
[  T641] >>[641] 4cf000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 4c5000 W 4210b6 55 8000000000000025
[  T641]  =[641] 4c5000 00000000350a4be7
[  T641]   [641] ->[639/1] 4c5000 4210b6
[  T639]   [641] <-[639/1] 4c5000 4000
[  T641] >>[641] 4c5000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 4cf000 W 4210d3 55 8000000000000025
[  T641]  =[641] 4cf000 00000000350a4be7
[  T641]   [641] ->[639/1] 4cf000 4210d3
[  T639]   [641] <-[639/1] 4cf000 4000
[  T641] >>[641] 4cf000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 4c2000 R 4488c0 54 0
[  T641]  =[641] 4c2000 00000000350a4be7
[  T641]   [641] ->[639/1] 4c2000 4488c0
[  T639]   [641] <-[639/1] 4c2000 0
[  T641] >>[641] 4c2000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 41b000 R 41b750 154 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## PAGEFAULT [641] 7ffff7ff9000 W 44dd09 55 0
[  T641]  =[641] 7ffff7ff9000 00000000350a4be7
[  T641]   [641] ->[639/1] 7ffff7ff9000 44dd09
[  T639]   [641] <-[639/1] 7ffff7ff9000 0
[  T641] >>[641] 7ffff7ff9000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 4ce000 W 403b16 55 8000000000000025
[  T642] 
[  T642] ## PAGEFAULT [642] 45d000 R 45dda0 154 0
[  T641]  =[641] 4ce000 00000000350a4be7
[  T642]   [642] VM_EXEC. continue
[  T641]   [641] ->[639/1] 4ce000 403b16
[  T642] 
[  T642] ## PAGEFAULT [642] 45d000 R 45dda0 170 0
[  T639]   [641] <-[639/1] 4ce000 4000
[  T642]   [642] VM_EXEC. continue
[  T641] >>[641] 4ce000 00000000350a4be7
[  T642] 
[  T642] ## PAGEFAULT [642] 495000 R 495390 154 0
[  T642]   [642] VM_EXEC. continue
[  T641] 
[  T641] ## VMA mmap [641] 0 - 801000, 0 20022
[  T642] 
[  T642] ## PAGEFAULT [642] 495000 R 495390 170 0
[  T642]   [642] VM_EXEC. continue
[  T642] 
[  T642] ## VMAFAULT [642] 4b4ef8 4965ce
[  T642]   [642] 4b4000 ->[-1/-1]
[  T642] ------------[ cut here ]------------
[  T642] kernel BUG at kernel/popcorn/pcn_kmsg.c:81!
[  T642] invalid opcode: 0000 [#1] SMP NOPTI
[  T642] CPU: 0 PID: 642 Comm: github-100 Tainted: G           O      5.2.0-rc4-popcorn+ #5
[  T642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20191013_105130-anatol 04/01/2014
[  T642] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T642] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T642] RSP: 0000:ffffc90000233e50 EFLAGS: 00000282
[  T642] RAX: 0000000000000018 RBX: 00000000004b4000 RCX: 000000000000001a
[  T642] RDX: ffff888139f643c0 RSI: 00000000ffffffff RDI: 0000000000000007
[  T641]   [641] 0 7ffff6ff9000 -- 7ffff77fa000
[  T642] RBP: ffff8881398c0100 R08: 0000000db72518d6 R09: 0000000000000000
[  T641]   [641] map+ 7ffff6ff9000 7ffff77fa000
[  T641]   [641] map0 7ffff6ff9000 -- 7ffff77fa000 @ 0, 0
[  T642] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001a
[  T642] R13: 00000000ffffffff R14: ffff88813783ea30 R15: ffff88813783ea78
[  T642] FS:  00007ffff7ffa700(0000) GS:ffff88813b600000(0000) knlGS:0000000000000000
[  T642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T642] CR2: 00000000004b4ef8 CR3: 0000000137a1c000 CR4: 00000000000006f0
[  T642] Call Trace:
[  T641] 
[  T641] VMA mprotect [641] 7ffff6ffa000 800000 3
[  T642]  vma_server_fetch_vma+0x27a/0x4b0
[  T642]  ? __do_page_fault+0x166/0x560
[  T642]  ? wait_woken+0x90/0x90
[  T642]  __do_page_fault+0x2ea/0x560
[  T642]  ? page_fault+0x8/0x30
[  T642]  page_fault+0x1e/0x30
[  T642] RIP: 0033:0x4965ce
[  T642] Code: 1c 24 77 02 19 c0 48 83 c4 18 5b 5d 41 5c 41 5d c3 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 41 56 41 55 41 54 55 53 48 83 ec 28 <8b> 16 85 d2 0f 84 0a 01 00 00 48 8d 44 24 18 45
[  T642] RSP: 002b:00007ffff7ff97b0 EFLAGS: 00000202
[  T642] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[  T642] RDX: 0000000000000000 RSI: 00000000004b4ef8 RDI: 00000000004c7380
[  T642] RBP: 00000000004b4ef8 R08: 00007ffff7ffa700 R09: 00007ffff7ffa700
[  T642] R10: 00007ffff7ffa9d0 R11: 0000000000000202 R12: 00007ffff7ff9bf8
[  T642] R13: 00000000004c7380 R14: 0000000000405be0 R15: 00007ffff7ffa700
[  T642] Modules linked in: msg_socket(O)
[  T642] ---[ end trace 98195f85b26ccfa5 ]---
[  T642] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T642] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T641]   [641] 0 7ffff6ffa000 -- 7ffff77fa000 3
[  T642] RSP: 0000:ffffc90000233e50 EFLAGS: 00000282
[  T641] 
[  T641] ## PAGEFAULT [641] 7ffff77f9000 W 403f53 55 0
[  T641]  =[641] 7ffff77f9000 00000000350a4be7
[  T642] RAX: 0000000000000018 RBX: 00000000004b4000 RCX: 000000000000001a
[  T641]   [641] ->[639/1] 7ffff77f9000 403f53
[  T642] RDX: ffff888139f643c0 RSI: 00000000ffffffff RDI: 0000000000000007
[  T642] RBP: ffff8881398c0100 R08: 0000000db72518d6 R09: 0000000000000000
[  T642] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001a
[  T642] R13: 00000000ffffffff R14: ffff88813783ea30 R15: ffff88813783ea78
[  T642] FS:  00007ffff7ffa700(0000) GS:ffff88813b600000(0000) knlGS:0000000000000000
[  T642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T642] CR2: 00000000004b4ef8 CR3: 0000000137a1c000 CR4: 00000000000006f0
[  T642] EXITED [642] remote / 0xb
[  T642] ------------[ cut here ]------------
[  T642] kernel BUG at kernel/popcorn/pcn_kmsg.c:81!
[  T642] invalid opcode: 0000 [#2] SMP NOPTI
[  T642] CPU: 1 PID: 642 Comm: github-100 Tainted: G      D    O      5.2.0-rc4-popcorn+ #5
[  T642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20191013_105130-anatol 04/01/2014
[  T642] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T642] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T642] RSP: 0018:ffffc90000233e88 EFLAGS: 00000282
[  T642] RAX: 000000000000000b RBX: ffff88813984a200 RCX: 0000000000000016
[  T642] RDX: ffffc90000233eaa RSI: 00000000ffffffff RDI: 0000000000000006
[  T642] RBP: ffff88813783e000 R08: ffff88813bbe63f8 R09: 0000000000000000
[  T642] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000016
[  T642] R13: 00000000ffffffff R14: ffff88813984a200 R15: 0000000000000000
[  T642] FS:  0000000000000000(0000) GS:ffff88813ba00000(0000) knlGS:0000000000000000
[  T642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T642] CR2: 00007ffff5f67010 CR3: 0000000137a1c000 CR4: 00000000000006e0
[  T642] Call Trace:
[  T642]  process_server_task_exit.cold+0xd2/0x335
[  T642]  do_exit+0x17c/0xba0
[  T642]  rewind_stack_do_exit+0x17/0x20
[  T642] Modules linked in: msg_socket(O)
[  T642] ---[ end trace 98195f85b26ccfa6 ]---
[  T642] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T642] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T642] RSP: 0000:ffffc90000233e50 EFLAGS: 00000282
[  T642] RAX: 0000000000000018 RBX: 00000000004b4000 RCX: 000000000000001a
[  T642] RDX: ffff888139f643c0 RSI: 00000000ffffffff RDI: 0000000000000007
[  T642] RBP: ffff8881398c0100 R08: 0000000db72518d6 R09: 0000000000000000
[  T642] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001a
[  T642] R13: 00000000ffffffff R14: ffff88813783ea30 R15: ffff88813783ea78
[  T642] FS:  0000000000000000(0000) GS:ffff88813ba00000(0000) knlGS:0000000000000000
[  T642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T642] CR2: 00007ffff5f67010 CR3: 0000000137a1c000 CR4: 00000000000006e0
[  T642] Fixing recursive fault but reboot is needed!
[  T639]   [641] <-[639/1] 7ffff77f9000 0
[  T641] >>[641] 7ffff77f9000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 7ffff77f8000 W 44dd09 55 0
[  T641]  =[641] 7ffff77f8000 00000000350a4be7
[  T641]   [641] ->[639/1] 7ffff77f8000 44dd09
[  T639]   [641] <-[639/1] 7ffff77f8000 0
[  T641] >>[641] 7ffff77f8000 00000000350a4be7
[  T643] ------------[ cut here ]------------
[  T643] kernel BUG at kernel/popcorn/pcn_kmsg.c:81!
[  T643] invalid opcode: 0000 [#3] SMP NOPTI
[  T643] CPU: 0 PID: 643 Comm: github-100 Tainted: G      D    O      5.2.0-rc4-popcorn+ #5
[  T643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20191013_105130-anatol 04/01/2014
[  T643] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T643] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T643] RSP: 0018:ffffc9000023bda8 EFLAGS: 00000282
[  T643] RAX: ffff888137a20240 RBX: 0000000000000000 RCX: 0000000000000042
[  T643] RDX: ffffc9000023bde6 RSI: 00000000ffffffff RDI: 0000000000000013
[  T643] RBP: ffffc9000023be58 R08: 0000000000000001 R09: ffff888137a20240
[  T643] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000042
[  T643] R13: 00000000ffffffff R14: 00000000004ccd20 R15: 0000000000000080
[  T643] FS:  00007ffff77f9700(0000) GS:ffff88813b600000(0000) knlGS:0000000000000000
[  T643] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T643] CR2: 00007ffff5f6b000 CR3: 0000000137a1c000 CR4: 00000000000006f0
[  T643] Call Trace:
[  T643]  process_server_do_futex_at_remote+0x103/0x140
[  T643]  __x64_sys_futex+0x1a8/0x1ba
[  T643]  do_syscall_64+0x69/0x440
[  T643]  ? trace_hardirqs_off_thunk+0x1a/0x1c
[  T643]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[  T643] RIP: 0033:0x40639f
[  T643] Code: 07 41 89 f0 83 f8 02 74 0b b8 02 00 00 00 87 07 85 c0 74 1f 44 89 c6 45 31 d2 ba 02 00 00 00 b8 ca 00 00 00 40 80 f6 80 0f 05 <eb> dd 0f 1f 80 00 00 00 00 c3 0f 1f 80 00 00 08
[  T643] RSP: 002b:00007ffff77f8878 EFLAGS: 00000282 ORIG_RAX: 00000000000000ca
[  T643] RAX: ffffffffffffffda RBX: 00000000004953c9 RCX: 000000000040639f
[  T643] RDX: 0000000000000002 RSI: 0000000000000080 RDI: 00000000004ccd20
[  T643] RBP: 00007ffff77f8990 R08: 0000000000000000 R09: 00007ffff77f9700
[  T643] R10: 0000000000000000 R11: 0000000000000282 R12: 00007ffff77f8bf8
[  T643] R13: 00007ffff77f8b50 R14: 0000000000405be0 R15: 00007ffff77f9700
[  T643] Modules linked in: msg_socket(O)
[  T643] ---[ end trace 98195f85b26ccfa7 ]---
[  T643] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T643] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T643] RSP: 0000:ffffc90000233e50 EFLAGS: 00000282
[  T643] RAX: 0000000000000018 RBX: 00000000004b4000 RCX: 000000000000001a
[  T643] RDX: ffff888139f643c0 RSI: 00000000ffffffff RDI: 0000000000000007
[  T643] RBP: ffff8881398c0100 R08: 0000000db72518d6 R09: 0000000000000000
[  T643] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001a
[  T643] R13: 00000000ffffffff R14: ffff88813783ea30 R15: ffff88813783ea78
[  T643] FS:  00007ffff77f9700(0000) GS:ffff88813b600000(0000) knlGS:0000000000000000
[  T643] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T643] CR2: 00007ffff5f6b000 CR3: 0000000137a1c000 CR4: 00000000000006f0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [640] 7ffff77f9000 R 401efd from [639/1]
[   T80] >>[640] 7ffff77f9000 00000000350a4be7
[   T80]   [640] ->[639/1] 0
[  T643] 
[  T643] ## PAGEFAULT [643] 7ffff77f9000 W 40639f 15 8000000000000065
[  T643]  =[643] 7ffff77f9000 00000000350a4be7
[  T643]   [643] ->[-1/-1] 7ffff77f9000 40639f
[  T643] ------------[ cut here ]------------
[  T643] kernel BUG at kernel/popcorn/pcn_kmsg.c:81!
[  T643] invalid opcode: 0000 [#4] SMP NOPTI
[  T643] CPU: 1 PID: 643 Comm: github-100 Tainted: G      D    O      5.2.0-rc4-popcorn+ #5
[  T643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20191013_105130-anatol 04/01/2014
[  T643] RIP: 0010:pcn_kmsg_post+0x7d/0x90
[  T643] Code: 88 45 00 e8 c5 05 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 3d e7 b7 01 5d 41 5c 41 5d 48 8b 40 28 e9 57 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T643] RSP: 0018:ffffc9000023bba0 EFLAGS: 00000282
[  T643] RAX: 0000000000000025 RBX: 00000000ffffffff RCX: 000000000000003a
[  T643] RDX: ffff888139f2a380 RSI: 00000000ffffffff RDI: 000000000000000b
[  T643] RBP: ffff888139f2a380 R08: 0000000000000001 R09: 0000000000000000
[  T643] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000003a
[  T643] R13: 00000000ffffffff R14: 0000000000000015 R15: 0000000000000002
[  T643] FS:  0000000000000000(0000) GS:ffff88813ba00000(0000) knlGS:0000000000000000
[  T643] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T643] CR2: 00007ffff77f99d0 CR3: 0000000137a1c000 CR4: 00000000000006e0
[  T643] Call Trace:
[  T643]  __request_remote_page.cold+0x46/0x65
[  T643]  __handle_localfault_at_remote+0x27c/0x5a4
[  T643]  page_server_handle_pte_fault.cold+0x191/0x889
[  T643]  ? do_trap+0x8b/0x110
[  T643]  ? pcn_kmsg_send+0x7d/0x90
[  T643]  __handle_mm_fault+0x2fa/0xb80
[  T643]  ? invalid_op+0x14/0x20
[  T643]  handle_mm_fault+0x6d/0xe6
[  T643]  __do_page_fault+0x273/0x560
[  T643]  page_fault+0x1e/0x30
[  T643] RIP: 0010:__put_user_4+0x1c/0x30
[  T643] Code: 66 90 c3 66 66 2e 0f 1f 84 00 00 00 00 00 65 48 8b 1c 25 00 5d 01 00 48 8b 9b 18 16 00 00 48 83 eb 03 48 39 d9 73 3c 66 66 90 <89> 01 31 c0 66 66 90 c3 66 66 2e 0f 1f 84 00 08
[  T643] RSP: 0018:ffffc9000023bec0 EFLAGS: 00000293
[  T643] RAX: 0000000000000000 RBX: 00007fffffffeffd RCX: 00007ffff77f99d0
[  T643] RDX: 0000000000000002 RSI: 0000000000000000 RDI: 00007ffff77f99d0
[  T643] RBP: ffff888137a20240 R08: 0000000000000001 R09: 0000000000000000
[  T643] R10: 0000000000000000 R11: 0000000000000000 R12: ffff888139aea000
[  T643] R13: 0000000000000000 R14: ffff888137a20240 R15: 0000000000000000
[  T643]  mm_release+0xe1/0x130
[  T643]  do_exit+0x15c/0xba0
[  T643]  ? trace_hardirqs_off_thunk+0x1a/0x1c
[  T643]  rewind_stack_do_exit+0x17/0x20
[  T643] Modules linked in: msg_socket(O)
[  T643] ---[ end trace 98195f85b26ccfa8 ]---
[  T643] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T643] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T643] RSP: 0000:ffffc90000233e50 EFLAGS: 00000282
[  T643] RAX: 0000000000000018 RBX: 00000000004b4000 RCX: 000000000000001a
[  T643] RDX: ffff888139f643c0 RSI: 00000000ffffffff RDI: 0000000000000007
[  T643] RBP: ffff8881398c0100 R08: 0000000db72518d6 R09: 0000000000000000
[  T643] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001a
[  T643] R13: 00000000ffffffff R14: ffff88813783ea30 R15: ffff88813783ea78
[  T643] FS:  0000000000000000(0000) GS:ffff88813ba00000(0000) knlGS:0000000000000000
[  T643] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T643] CR2: 00007ffff77f99d0 CR3: 0000000137a1c000 CR4: 00000000000006e0
[  T643] Fixing recursive fault but reboot is needed!

Log on host 1 after program has hung bug before ps aux is invoked a

x86 login: [  T635] msg_socket: loading out-of-tree module taints kernel.
[  T635] Loading Popcorn messaging layer over TCP/IP...
[  T635] popcorn: Loading node configuration...
[  T635] popcorn:   0: 10.4.4.200
[  T635] popcorn: * 1: 10.4.4.201
[  T635] Ready to accept incoming connections
[  T635] Connecting to 0 at 10.4.4.200
[  T637] SEND handler for 0 is ready
[  T638] RECV handler for 0 is ready
[  T638] popcorn:    0 joined, x86_64
[  T635] popcorn: Ready on TCP/IP
[  T639] ####### MIGRATE [639] to 0
[  T639] 
[  T639] ## VMA_INFO [639] 401000 -- 499000 875
[  T639]   [639] /home/popcorn/github-100 + 1
[  T639] 
[  T639] ## VMA_INFO [639] 7ffffffde000 -- 7ffffffff000 100173
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 7fffffffe000 R 401f00 from [641/0]
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 7fffffffe000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[  T639] 
[  T639] ## VMA_INFO [639] 4c8000 -- 4f1000 100073
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 4ce000 R 401fd7 from [641/0]
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 4ce000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 7fffffffe000 W 401cbd from [641/0]
[   T80]  =[639] origin remote 000000005810dd2b
[   T80] >>[639] 7fffffffe000 000000005810dd2b
[   T80]   [639] ->[641/0] 4000
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 4cb000 W 4035fc from [641/0]
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 4cb000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 4cc000 R 40360a from [641/0]
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 4cc000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[  T639] 
[  T639] ## VMA_INFO [639] 4c2000 -- 4c8000 100873
[  T639]   [639] /home/popcorn/github-100 + c1
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 4c6000 R 44c924 from [641/0]
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 4c6000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 4c7000 W 40376a from [641/0]
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 4c7000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 4c5000 R 40377c from [641/0]
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 4c5000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[  T639] 
[  T639] VMA_OP_REQUEST [639] mmap 0 801000
[  T639]   [639] 0 7ffff77fa000 -- 7ffff7ffb000 0 20022
[  T639]   [639] ->mmap 0
[  T639] 
[  T639] VMA_OP_REQUEST [639] mprotect 7ffff77fb000 800000
[  T639]   [639] ->mprotect 0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 7ffff7ffa000 W 403f53 from [641/0]
[   T80]   [639] handle local fault at origin
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 7ffff7ffa000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 4cd000 W 403f8f from [641/0]
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 4cd000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 4cc000 W 403fa0 from [641/0]
[   T80]  =[639] origin remote 000000005810dd2b
[   T80] >>[639] 4cc000 000000005810dd2b
[   T80]   [639] ->[641/0] 4000
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 4cf000 R 422ebd from [641/0]
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 4cf000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 4c5000 W 4210b6 from [641/0]
[   T80]  =[639] origin remote 000000005810dd2b
[   T80] >>[639] 4c5000 000000005810dd2b
[   T80]   [639] ->[641/0] 4000
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 4cf000 W 4210d3 from [641/0]
[   T80]  =[639] origin remote 000000005810dd2b
[   T80] >>[639] 4cf000 000000005810dd2b
[   T80]   [639] ->[641/0] 4000
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 4c2000 R 4488c0 from [641/0]
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 4c2000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 7ffff7ff9000 W 44dd09 from [641/0]
[   T80]   [639] handle local fault at origin
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 7ffff7ff9000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 4ce000 W 403b16 from [641/0]
[   T80]  =[639] origin remote 000000005810dd2b
[   T80] >>[639] 4ce000 000000005810dd2b
[   T80]   [639] ->[641/0] 4000
[  T639] 
[  T639] VMA_OP_REQUEST [639] mmap 0 801000
[  T639]   [639] 0 7ffff6ff9000 -- 7ffff77fa000 0 20022
[  T639]   [639] ->mmap 0
[  T639] 
[  T639] VMA_OP_REQUEST [639] mprotect 7ffff6ffa000 800000
[  T639]   [639] ->mprotect 0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 7ffff77f9000 W 403f53 from [641/0]
[   T80]   [639] handle local fault at origin
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 7ffff77f9000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [639] 7ffff77f8000 W 44dd09 from [641/0]
[   T80]   [639] handle local fault at origin
[   T80]  =[639] origin  000000005810dd2b
[   T80] >>[639] 7ffff77f8000 000000005810dd2b
[   T80]   [639] ->[641/0] 0
[  T639]   [639] ->[640/0] 7ffff77f9000 401efd
[  T638]   [639] <-[640/0] 7ffff77f9000 0
[  T639] 
[  T639] ## PAGEFAULT [639] 7ffff77f9000 W 401efd 5 8000000000000065
[  T639]   [639] waits 000000005810dd2b

Additional log on host 1 after ps aux is invoked and then segfaults

[  OK  ] Started Getty on tty1.
         Starting Serial Getty on ttyS0...
[  OK  ] Started Serial Getty on ttyS0.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started System Logging Service.

Debian GNU/Linux 8 x86 ttyS0

x86 login: [  T636] msg_socket: loading out-of-tree module taints kernel.
[  T636] Loading Popcorn messaging layer over TCP/IP...
[  T636] popcorn: Loading node configuration...
[  T636] popcorn: * 0: 10.4.4.200
[  T636] popcorn:   1: 10.4.4.201
[  T636] Ready to accept incoming connections
[  T638] SEND handler for 1 is ready
[  T639] RECV handler for 1 is ready
[  T636] popcorn: Ready on TCP/IP
[  T639] popcorn:    1 joined, x86_64
[  T640] remote_worker_main: [640] for [639/1]
[  T640] remote_worker_main: [640] /home/popcorn/github-100
[  T641] remote_thread_main [641] started for [639/1]
[  T641] restore_thread_info [641] ip 401efd
[  T641] restore_thread_info [641] sp 7fffffffe2d0 bp 408290
[  T641] 
[  T641] ####### MIGRATED - [641/0] from [639/1]
[  T641] 
[  T641] ## VMAFAULT [641] 401efd 401efd
[  T641]   [641] 401000 ->[639/1]
[  T641]   [641] 0 401000
[  T641]   [641] /home/popcorn/github-100 + 1
[  T641]   [641] map+ 401000 499000
[  T641]   [641] map0 401000 -- 499000 @ 1, 5
[  T641] 
[  T641] ## PAGEFAULT [641] 401000 R 401efd 154 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## PAGEFAULT [641] 401000 R 401efd 170 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## VMAFAULT [641] 7fffffffe308 401f00
[  T641]   [641] 7fffffffe000 ->[639/1]
[  T641]   [641] 0 7fffffffe000
[  T641]   [641] map+ 7ffffffde000 7ffffffff000
[  T641]   [641] map0 7ffffffde000 -- 7ffffffff000 @ 7ffffffde, 3
[  T641] 
[  T641] ## PAGEFAULT [641] 7fffffffe000 R 401f00 54 0
[  T641]  =[641] 7fffffffe000 00000000350a4be7
[  T641]   [641] ->[639/1] 7fffffffe000 401f00
[  T639]   [641] <-[639/1] 7fffffffe000 0
[  T641] >>[641] 7fffffffe000 00000000350a4be7
[  T641] 
[  T641] ## VMAFAULT [641] 4ce8a8 401fd7
[  T641]   [641] 4ce000 ->[639/1]
[  T641]   [641] 7ffffffde000 4ce000
[  T641]   [641] map+ 4c8000 4f1000
[  T641]   [641] map0 4c8000 -- 4f1000 @ 4c8, 3
[  T641] 
[  T641] ## PAGEFAULT [641] 4ce000 R 401fd7 54 0
[  T641]  =[641] 4ce000 00000000350a4be7
[  T641]   [641] ->[639/1] 4ce000 401fd7
[  T639]   [641] <-[639/1] 4ce000 0
[  T641] >>[641] 4ce000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 7fffffffe000 W 401cbd 55 8000000000000025
[  T641]  =[641] 7fffffffe000 00000000350a4be7
[  T641]   [641] ->[639/1] 7fffffffe000 401cbd
[  T639]   [641] <-[639/1] 7fffffffe000 4000
[  T641] >>[641] 7fffffffe000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 4cb000 W 4035fc 55 0
[  T641]  =[641] 4cb000 00000000350a4be7
[  T641]   [641] ->[639/1] 4cb000 4035fc
[  T639]   [641] <-[639/1] 4cb000 0
[  T641] >>[641] 4cb000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 4cc000 R 40360a 54 0
[  T641]  =[641] 4cc000 00000000350a4be7
[  T641]   [641] ->[639/1] 4cc000 40360a
[  T639]   [641] <-[639/1] 4cc000 0
[  T641] >>[641] 4cc000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 44c000 R 44c920 154 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## PAGEFAULT [641] 44c000 R 44c920 170 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## VMAFAULT [641] 4c6198 44c924
[  T641]   [641] 4c6000 ->[639/1]
[  T641]   [641] 4c8000 4c6000
[  T641]   [641] /home/popcorn/github-100 + c1
[  T641]   [641] map+ 4c2000 4c8000
[  T641]   [641] map0 4c2000 -- 4c8000 @ c1, 3
[  T641] 
[  T641] ## PAGEFAULT [641] 4c6000 R 44c924 54 0
[  T641]  =[641] 4c6000 0000000025699d3b
[  T641]   [641] ->[639/1] 4c6000 44c924
[  T639]   [641] <-[639/1] 4c6000 0
[  T641] >>[641] 4c6000 0000000025699d3b
[  T641] 
[  T641] ## PAGEFAULT [641] 4c7000 W 40376a 55 0
[  T641]  =[641] 4c7000 0000000025699d3b
[  T641]   [641] ->[639/1] 4c7000 40376a
[  T639]   [641] <-[639/1] 4c7000 0
[  T641] >>[641] 4c7000 0000000025699d3b
[  T641] 
[  T641] ## PAGEFAULT [641] 4c5000 R 40377c 54 0
[  T641]  =[641] 4c5000 0000000025699d3b
[  T641]   [641] ->[639/1] 4c5000 40377c
[  T639]   [641] <-[639/1] 4c5000 0
[  T641] >>[641] 4c5000 0000000025699d3b
[  T641] 
[  T641] ## VMA mmap [641] 0 - 801000, 0 20022
[  T641]   [641] 0 7ffff77fa000 -- 7ffff7ffb000
[  T641]   [641] map+ 7ffff77fa000 7ffff7ffb000
[  T641]   [641] map0 7ffff77fa000 -- 7ffff7ffb000 @ 0, 0
[  T641] 
[  T641] VMA mprotect [641] 7ffff77fb000 800000 3
[  T641]   [641] 0 7ffff77fb000 -- 7ffff7ffb000 3
[  T641] 
[  T641] ## PAGEFAULT [641] 7ffff7ffa000 W 403f53 55 0
[  T641]  =[641] 7ffff7ffa000 0000000025699d3b
[  T641]   [641] ->[639/1] 7ffff7ffa000 403f53
[  T639]   [641] <-[639/1] 7ffff7ffa000 0
[  T641] >>[641] 7ffff7ffa000 0000000025699d3b
[  T641] 
[  T641] ## PAGEFAULT [641] 4cd000 W 403f8f 55 0
[  T641]  =[641] 4cd000 0000000025699d3b
[  T641]   [641] ->[639/1] 4cd000 403f8f
[  T639]   [641] <-[639/1] 4cd000 0
[  T641] >>[641] 4cd000 0000000025699d3b
[  T641] 
[  T641] ## PAGEFAULT [641] 4cc000 W 403fa0 55 8000000000000025
[  T641]  =[641] 4cc000 0000000025699d3b
[  T641]   [641] ->[639/1] 4cc000 403fa0
[  T639]   [641] <-[639/1] 4cc000 4000
[  T641] >>[641] 4cc000 0000000025699d3b
[  T641] 
[  T641] ## PAGEFAULT [641] 422000 R 422e50 154 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## PAGEFAULT [641] 422000 R 422e50 170 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## PAGEFAULT [641] 4cf000 R 422ebd 54 0
[  T641]  =[641] 4cf000 00000000350a4be7
[  T641]   [641] ->[639/1] 4cf000 422ebd
[  T639]   [641] <-[639/1] 4cf000 0
[  T641] >>[641] 4cf000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 4c5000 W 4210b6 55 8000000000000025
[  T641]  =[641] 4c5000 00000000350a4be7
[  T641]   [641] ->[639/1] 4c5000 4210b6
[  T639]   [641] <-[639/1] 4c5000 4000
[  T641] >>[641] 4c5000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 4cf000 W 4210d3 55 8000000000000025
[  T641]  =[641] 4cf000 00000000350a4be7
[  T641]   [641] ->[639/1] 4cf000 4210d3
[  T639]   [641] <-[639/1] 4cf000 4000
[  T641] >>[641] 4cf000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 4c2000 R 4488c0 54 0
[  T641]  =[641] 4c2000 00000000350a4be7
[  T641]   [641] ->[639/1] 4c2000 4488c0
[  T639]   [641] <-[639/1] 4c2000 0
[  T641] >>[641] 4c2000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 41b000 R 41b750 154 0
[  T641]   [641] VM_EXEC. continue
[  T641] 
[  T641] ## PAGEFAULT [641] 7ffff7ff9000 W 44dd09 55 0
[  T641]  =[641] 7ffff7ff9000 00000000350a4be7
[  T641]   [641] ->[639/1] 7ffff7ff9000 44dd09
[  T639]   [641] <-[639/1] 7ffff7ff9000 0
[  T641] >>[641] 7ffff7ff9000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 4ce000 W 403b16 55 8000000000000025
[  T642] 
[  T642] ## PAGEFAULT [642] 45d000 R 45dda0 154 0
[  T641]  =[641] 4ce000 00000000350a4be7
[  T642]   [642] VM_EXEC. continue
[  T641]   [641] ->[639/1] 4ce000 403b16
[  T642] 
[  T642] ## PAGEFAULT [642] 45d000 R 45dda0 170 0
[  T639]   [641] <-[639/1] 4ce000 4000
[  T642]   [642] VM_EXEC. continue
[  T641] >>[641] 4ce000 00000000350a4be7
[  T642] 
[  T642] ## PAGEFAULT [642] 495000 R 495390 154 0
[  T642]   [642] VM_EXEC. continue
[  T641] 
[  T641] ## VMA mmap [641] 0 - 801000, 0 20022
[  T642] 
[  T642] ## PAGEFAULT [642] 495000 R 495390 170 0
[  T642]   [642] VM_EXEC. continue
[  T642] 
[  T642] ## VMAFAULT [642] 4b4ef8 4965ce
[  T642]   [642] 4b4000 ->[-1/-1]
[  T642] ------------[ cut here ]------------
[  T642] kernel BUG at kernel/popcorn/pcn_kmsg.c:81!
[  T642] invalid opcode: 0000 [#1] SMP NOPTI
[  T642] CPU: 0 PID: 642 Comm: github-100 Tainted: G           O      5.2.0-rc4-popcorn+ #5
[  T642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20191013_105130-anatol 04/01/2014
[  T642] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T642] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T642] RSP: 0000:ffffc90000233e50 EFLAGS: 00000282
[  T642] RAX: 0000000000000018 RBX: 00000000004b4000 RCX: 000000000000001a
[  T642] RDX: ffff888139f643c0 RSI: 00000000ffffffff RDI: 0000000000000007
[  T641]   [641] 0 7ffff6ff9000 -- 7ffff77fa000
[  T642] RBP: ffff8881398c0100 R08: 0000000db72518d6 R09: 0000000000000000
[  T641]   [641] map+ 7ffff6ff9000 7ffff77fa000
[  T641]   [641] map0 7ffff6ff9000 -- 7ffff77fa000 @ 0, 0
[  T642] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001a
[  T642] R13: 00000000ffffffff R14: ffff88813783ea30 R15: ffff88813783ea78
[  T642] FS:  00007ffff7ffa700(0000) GS:ffff88813b600000(0000) knlGS:0000000000000000
[  T642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T642] CR2: 00000000004b4ef8 CR3: 0000000137a1c000 CR4: 00000000000006f0
[  T642] Call Trace:
[  T641] 
[  T641] VMA mprotect [641] 7ffff6ffa000 800000 3
[  T642]  vma_server_fetch_vma+0x27a/0x4b0
[  T642]  ? __do_page_fault+0x166/0x560
[  T642]  ? wait_woken+0x90/0x90
[  T642]  __do_page_fault+0x2ea/0x560
[  T642]  ? page_fault+0x8/0x30
[  T642]  page_fault+0x1e/0x30
[  T642] RIP: 0033:0x4965ce
[  T642] Code: 1c 24 77 02 19 c0 48 83 c4 18 5b 5d 41 5c 41 5d c3 66 66 2e 0f 1f 84 00 00 00 00 00 41 57 41 56 41 55 41 54 55 53 48 83 ec 28 <8b> 16 85 d2 0f 84 0a 01 00 00 48 8d 44 24 18 45
[  T642] RSP: 002b:00007ffff7ff97b0 EFLAGS: 00000202
[  T642] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[  T642] RDX: 0000000000000000 RSI: 00000000004b4ef8 RDI: 00000000004c7380
[  T642] RBP: 00000000004b4ef8 R08: 00007ffff7ffa700 R09: 00007ffff7ffa700
[  T642] R10: 00007ffff7ffa9d0 R11: 0000000000000202 R12: 00007ffff7ff9bf8
[  T642] R13: 00000000004c7380 R14: 0000000000405be0 R15: 00007ffff7ffa700
[  T642] Modules linked in: msg_socket(O)
[  T642] ---[ end trace 98195f85b26ccfa5 ]---
[  T642] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T642] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T641]   [641] 0 7ffff6ffa000 -- 7ffff77fa000 3
[  T642] RSP: 0000:ffffc90000233e50 EFLAGS: 00000282
[  T641] 
[  T641] ## PAGEFAULT [641] 7ffff77f9000 W 403f53 55 0
[  T641]  =[641] 7ffff77f9000 00000000350a4be7
[  T642] RAX: 0000000000000018 RBX: 00000000004b4000 RCX: 000000000000001a
[  T641]   [641] ->[639/1] 7ffff77f9000 403f53
[  T642] RDX: ffff888139f643c0 RSI: 00000000ffffffff RDI: 0000000000000007
[  T642] RBP: ffff8881398c0100 R08: 0000000db72518d6 R09: 0000000000000000
[  T642] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001a
[  T642] R13: 00000000ffffffff R14: ffff88813783ea30 R15: ffff88813783ea78
[  T642] FS:  00007ffff7ffa700(0000) GS:ffff88813b600000(0000) knlGS:0000000000000000
[  T642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T642] CR2: 00000000004b4ef8 CR3: 0000000137a1c000 CR4: 00000000000006f0
[  T642] EXITED [642] remote / 0xb
[  T642] ------------[ cut here ]------------
[  T642] kernel BUG at kernel/popcorn/pcn_kmsg.c:81!
[  T642] invalid opcode: 0000 [#2] SMP NOPTI
[  T642] CPU: 1 PID: 642 Comm: github-100 Tainted: G      D    O      5.2.0-rc4-popcorn+ #5
[  T642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20191013_105130-anatol 04/01/2014
[  T642] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T642] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T642] RSP: 0018:ffffc90000233e88 EFLAGS: 00000282
[  T642] RAX: 000000000000000b RBX: ffff88813984a200 RCX: 0000000000000016
[  T642] RDX: ffffc90000233eaa RSI: 00000000ffffffff RDI: 0000000000000006
[  T642] RBP: ffff88813783e000 R08: ffff88813bbe63f8 R09: 0000000000000000
[  T642] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000016
[  T642] R13: 00000000ffffffff R14: ffff88813984a200 R15: 0000000000000000
[  T642] FS:  0000000000000000(0000) GS:ffff88813ba00000(0000) knlGS:0000000000000000
[  T642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T642] CR2: 00007ffff5f67010 CR3: 0000000137a1c000 CR4: 00000000000006e0
[  T642] Call Trace:
[  T642]  process_server_task_exit.cold+0xd2/0x335
[  T642]  do_exit+0x17c/0xba0
[  T642]  rewind_stack_do_exit+0x17/0x20
[  T642] Modules linked in: msg_socket(O)
[  T642] ---[ end trace 98195f85b26ccfa6 ]---
[  T642] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T642] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T642] RSP: 0000:ffffc90000233e50 EFLAGS: 00000282
[  T642] RAX: 0000000000000018 RBX: 00000000004b4000 RCX: 000000000000001a
[  T642] RDX: ffff888139f643c0 RSI: 00000000ffffffff RDI: 0000000000000007
[  T642] RBP: ffff8881398c0100 R08: 0000000db72518d6 R09: 0000000000000000
[  T642] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001a
[  T642] R13: 00000000ffffffff R14: ffff88813783ea30 R15: ffff88813783ea78
[  T642] FS:  0000000000000000(0000) GS:ffff88813ba00000(0000) knlGS:0000000000000000
[  T642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T642] CR2: 00007ffff5f67010 CR3: 0000000137a1c000 CR4: 00000000000006e0
[  T642] Fixing recursive fault but reboot is needed!
[  T639]   [641] <-[639/1] 7ffff77f9000 0
[  T641] >>[641] 7ffff77f9000 00000000350a4be7
[  T641] 
[  T641] ## PAGEFAULT [641] 7ffff77f8000 W 44dd09 55 0
[  T641]  =[641] 7ffff77f8000 00000000350a4be7
[  T641]   [641] ->[639/1] 7ffff77f8000 44dd09
[  T639]   [641] <-[639/1] 7ffff77f8000 0
[  T641] >>[641] 7ffff77f8000 00000000350a4be7
[  T643] ------------[ cut here ]------------
[  T643] kernel BUG at kernel/popcorn/pcn_kmsg.c:81!
[  T643] invalid opcode: 0000 [#3] SMP NOPTI
[  T643] CPU: 0 PID: 643 Comm: github-100 Tainted: G      D    O      5.2.0-rc4-popcorn+ #5
[  T643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20191013_105130-anatol 04/01/2014
[  T643] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T643] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T643] RSP: 0018:ffffc9000023bda8 EFLAGS: 00000282
[  T643] RAX: ffff888137a20240 RBX: 0000000000000000 RCX: 0000000000000042
[  T643] RDX: ffffc9000023bde6 RSI: 00000000ffffffff RDI: 0000000000000013
[  T643] RBP: ffffc9000023be58 R08: 0000000000000001 R09: ffff888137a20240
[  T643] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000042
[  T643] R13: 00000000ffffffff R14: 00000000004ccd20 R15: 0000000000000080
[  T643] FS:  00007ffff77f9700(0000) GS:ffff88813b600000(0000) knlGS:0000000000000000
[  T643] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T643] CR2: 00007ffff5f6b000 CR3: 0000000137a1c000 CR4: 00000000000006f0
[  T643] Call Trace:
[  T643]  process_server_do_futex_at_remote+0x103/0x140
[  T643]  __x64_sys_futex+0x1a8/0x1ba
[  T643]  do_syscall_64+0x69/0x440
[  T643]  ? trace_hardirqs_off_thunk+0x1a/0x1c
[  T643]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[  T643] RIP: 0033:0x40639f
[  T643] Code: 07 41 89 f0 83 f8 02 74 0b b8 02 00 00 00 87 07 85 c0 74 1f 44 89 c6 45 31 d2 ba 02 00 00 00 b8 ca 00 00 00 40 80 f6 80 0f 05 <eb> dd 0f 1f 80 00 00 00 00 c3 0f 1f 80 00 00 08
[  T643] RSP: 002b:00007ffff77f8878 EFLAGS: 00000282 ORIG_RAX: 00000000000000ca
[  T643] RAX: ffffffffffffffda RBX: 00000000004953c9 RCX: 000000000040639f
[  T643] RDX: 0000000000000002 RSI: 0000000000000080 RDI: 00000000004ccd20
[  T643] RBP: 00007ffff77f8990 R08: 0000000000000000 R09: 00007ffff77f9700
[  T643] R10: 0000000000000000 R11: 0000000000000282 R12: 00007ffff77f8bf8
[  T643] R13: 00007ffff77f8b50 R14: 0000000000405be0 R15: 00007ffff77f9700
[  T643] Modules linked in: msg_socket(O)
[  T643] ---[ end trace 98195f85b26ccfa7 ]---
[  T643] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T643] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T643] RSP: 0000:ffffc90000233e50 EFLAGS: 00000282
[  T643] RAX: 0000000000000018 RBX: 00000000004b4000 RCX: 000000000000001a
[  T643] RDX: ffff888139f643c0 RSI: 00000000ffffffff RDI: 0000000000000007
[  T643] RBP: ffff8881398c0100 R08: 0000000db72518d6 R09: 0000000000000000
[  T643] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001a
[  T643] R13: 00000000ffffffff R14: ffff88813783ea30 R15: ffff88813783ea78
[  T643] FS:  00007ffff77f9700(0000) GS:ffff88813b600000(0000) knlGS:0000000000000000
[  T643] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T643] CR2: 00007ffff5f6b000 CR3: 0000000137a1c000 CR4: 00000000000006f0
[   T80] 
[   T80] REMOTE_PAGE_REQUEST [640] 7ffff77f9000 R 401efd from [639/1]
[   T80] >>[640] 7ffff77f9000 00000000350a4be7
[   T80]   [640] ->[639/1] 0
[  T643] 
[  T643] ## PAGEFAULT [643] 7ffff77f9000 W 40639f 15 8000000000000065
[  T643]  =[643] 7ffff77f9000 00000000350a4be7
[  T643]   [643] ->[-1/-1] 7ffff77f9000 40639f
[  T643] ------------[ cut here ]------------
[  T643] kernel BUG at kernel/popcorn/pcn_kmsg.c:81!
[  T643] invalid opcode: 0000 [#4] SMP NOPTI
[  T643] CPU: 1 PID: 643 Comm: github-100 Tainted: G      D    O      5.2.0-rc4-popcorn+ #5
[  T643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20191013_105130-anatol 04/01/2014
[  T643] RIP: 0010:pcn_kmsg_post+0x7d/0x90
[  T643] Code: 88 45 00 e8 c5 05 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 3d e7 b7 01 5d 41 5c 41 5d 48 8b 40 28 e9 57 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T643] RSP: 0018:ffffc9000023bba0 EFLAGS: 00000282
[  T643] RAX: 0000000000000025 RBX: 00000000ffffffff RCX: 000000000000003a
[  T643] RDX: ffff888139f2a380 RSI: 00000000ffffffff RDI: 000000000000000b
[  T643] RBP: ffff888139f2a380 R08: 0000000000000001 R09: 0000000000000000
[  T643] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000003a
[  T643] R13: 00000000ffffffff R14: 0000000000000015 R15: 0000000000000002
[  T643] FS:  0000000000000000(0000) GS:ffff88813ba00000(0000) knlGS:0000000000000000
[  T643] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T643] CR2: 00007ffff77f99d0 CR3: 0000000137a1c000 CR4: 00000000000006e0
[  T643] Call Trace:
[  T643]  __request_remote_page.cold+0x46/0x65
[  T643]  __handle_localfault_at_remote+0x27c/0x5a4
[  T643]  page_server_handle_pte_fault.cold+0x191/0x889
[  T643]  ? do_trap+0x8b/0x110
[  T643]  ? pcn_kmsg_send+0x7d/0x90
[  T643]  __handle_mm_fault+0x2fa/0xb80
[  T643]  ? invalid_op+0x14/0x20
[  T643]  handle_mm_fault+0x6d/0xe6
[  T643]  __do_page_fault+0x273/0x560
[  T643]  page_fault+0x1e/0x30
[  T643] RIP: 0010:__put_user_4+0x1c/0x30
[  T643] Code: 66 90 c3 66 66 2e 0f 1f 84 00 00 00 00 00 65 48 8b 1c 25 00 5d 01 00 48 8b 9b 18 16 00 00 48 83 eb 03 48 39 d9 73 3c 66 66 90 <89> 01 31 c0 66 66 90 c3 66 66 2e 0f 1f 84 00 08
[  T643] RSP: 0018:ffffc9000023bec0 EFLAGS: 00000293
[  T643] RAX: 0000000000000000 RBX: 00007fffffffeffd RCX: 00007ffff77f99d0
[  T643] RDX: 0000000000000002 RSI: 0000000000000000 RDI: 00007ffff77f99d0
[  T643] RBP: ffff888137a20240 R08: 0000000000000001 R09: 0000000000000000
[  T643] R10: 0000000000000000 R11: 0000000000000000 R12: ffff888139aea000
[  T643] R13: 0000000000000000 R14: ffff888137a20240 R15: 0000000000000000
[  T643]  mm_release+0xe1/0x130
[  T643]  do_exit+0x15c/0xba0
[  T643]  ? trace_hardirqs_off_thunk+0x1a/0x1c
[  T643]  rewind_stack_do_exit+0x17/0x20
[  T643] Modules linked in: msg_socket(O)
[  T643] ---[ end trace 98195f85b26ccfa8 ]---
[  T643] RIP: 0010:pcn_kmsg_send+0x7d/0x90
[  T643] Code: 88 45 00 e8 55 06 00 00 4c 89 e2 48 89 ee 44 89 ef 48 8b 05 cd e7 b7 01 5d 41 5c 41 5d 48 8b 40 20 e9 e7 0f 67 00 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 06
[  T643] RSP: 0000:ffffc90000233e50 EFLAGS: 00000282
[  T643] RAX: 0000000000000018 RBX: 00000000004b4000 RCX: 000000000000001a
[  T643] RDX: ffff888139f643c0 RSI: 00000000ffffffff RDI: 0000000000000007
[  T643] RBP: ffff8881398c0100 R08: 0000000db72518d6 R09: 0000000000000000
[  T643] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001a
[  T643] R13: 00000000ffffffff R14: ffff88813783ea30 R15: ffff88813783ea78
[  T643] FS:  0000000000000000(0000) GS:ffff88813ba00000(0000) knlGS:0000000000000000
[  T643] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  T643] CR2: 00007ffff77f99d0 CR3: 0000000137a1c000 CR4: 00000000000006e0
[  T643] Fixing recursive fault but reboot is needed!
@beowulf
Copy link
Collaborator

beowulf commented Apr 2, 2020

Creating threads on remote is not supported yet.

@bxatnarf bxatnarf changed the title [upstream] execution hung on both nodes due to recursive fault on remote remote thread creation needs to fail more gently Apr 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@beowulf @bxatnarf