arbiter: Wait for all old workers to terminate on SIGHUP

This is to prevent them getting double SIGTERM signals sent to them
(once the main loop runs self.manage_workers again and notices that
there are too many workers spawned). It seems as if they exit due to
signal termination in case they are sent SIGTERM, but are not yet
reaped.

Hopefully solves #3050.

gunicorn/arbiter.py

@@ -478,13 +478,23 @@ def reload(self):
         # set new proc_name
         util._setproctitle("master [%s]" % self.proc_name)
 
+        last_worker_to_kill = self.worker_age
+
         # spawn new workers
         for _ in range(self.cfg.workers):
             self.spawn_worker()
 
         # manage workers
         self.manage_workers()
 
+        # wait for all old workers to be replaced
+        deadline = time.monotonic() + self.cfg.graceful_timeout
+        while time.monotonic() < deadline:
+            oldest_worker = list(self.WORKERS.values())[0].age
+            if oldest_worker > last_worker_to_kill:
+                break
+            time.sleep(0.1)
+
     def murder_workers(self):
         """\
         Kill unused/idle workers