Skip to content

Commit

Permalink
[inductor] subproc parallel compile: initialize future before sending…
Browse files Browse the repository at this point in the history
… work to the pool (pytorch#128086)

Summary: I got reports of intermittent failures in CI and the logs show errors like this:
```
CRITICAL:concurrent.futures:Future 139789013754560 in unexpected state: FINISHED
```
I can't repro locally, but seems clear that we should initialize the future _before_ sending work to the subprocess pool since it could finish before we call set_running_or_notify_cancel()

Differential Revision: [D58239829](https://our.internmc.facebook.com/intern/diff/D58239829)
Pull Request resolved: pytorch#128086
Approved by: https://github.com/jansel
ghstack dependencies: pytorch#128037
  • Loading branch information
masnesral authored and pytorchmergebot committed Jun 7, 2024
1 parent 6a2bf48 commit dc8e3c2
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion torch/_inductor/compile_worker/subproc_pool.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,11 +109,11 @@ def submit(self, job_fn: Callable[..., Any], *args):
with self.futures_lock:
job_id = next(self.job_id_count)
self.pending_futures[job_id] = future = Future()
future.set_running_or_notify_cancel()
with self.write_lock:
if not self.running:
raise RuntimeError("submit() on closed pool")
_send_msg(self.write_pipe, job_id, job_data)
future.set_running_or_notify_cancel()
return future

def _read_thread(self):
Expand Down

0 comments on commit dc8e3c2

Please sign in to comment.