Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove "--enable_pybind" from webgpu pipeline #23550

Merged
merged 1 commit into from
Jan 31, 2025
Merged

Remove "--enable_pybind" from webgpu pipeline #23550

merged 1 commit into from
Jan 31, 2025

Conversation

snnn
Copy link
Member

@snnn snnn commented Jan 31, 2025

There is a crash in the WebGPU CI pipeline. It crashed at process shutdown when unloading onnxruntime_pybind11_state.pyd.
Here is the callstack:

 	dxil.dll!DxcSwapThreadMalloc()	Unknown
 	dxil.dll!DxcThreadMalloc::DxcThreadMalloc(struct IMalloc *)	Unknown
 	dxil.dll!DxcValidator::Release(void)	Unknown
 	[Inline Frame] webgpu_dawn.dll!Microsoft::WRL::ComPtr<IDxcValidator>::InternalRelease() Line 235	C++
 	[Inline Frame] webgpu_dawn.dll!Microsoft::WRL::ComPtr<IDxcValidator>::{dtor}() Line 290	C++
 	webgpu_dawn.dll!dawn::native::d3d12::Backend::`scalar deleting destructor'(unsigned int)	C++
 	webgpu_dawn.dll!`eh vector destructor iterator'(void * ptr, unsigned __int64 size, unsigned __int64 count, void(*)(void *) destructor)	C++
 	webgpu_dawn.dll!dawn::native::InstanceBase::~InstanceBase() Line 197	C++
 	webgpu_dawn.dll!dawn::native::InstanceBase::`scalar deleting destructor'(unsigned int)	C++
 	webgpu_dawn.dll!dawn::native::InstanceBase::DeleteThis() Line 218	C++
 	ucrtbase.dll!<lambda>(void)()	Unknown
 	ucrtbase.dll!__crt_seh_guarded_call<int>::operator()<<lambda_7777bce6b2f8c936911f934f8298dc43>,<lambda>(void) &,<lambda_3883c3dff614d5e0c5f61bb1ac94921c>>()	Unknown
 	ucrtbase.dll!_execute_onexit_table()	Unknown
 	onnxruntime_pybind11_state.pyd!dllmain_crt_process_detach(const bool is_terminating) Line 182	C++
>	onnxruntime_pybind11_state.pyd!dllmain_dispatch(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 293	C++
 	ntdll.dll!LdrpCallInitRoutine()	Unknown
 	ntdll.dll!LdrShutdownProcess()	Unknown
 	ntdll.dll!RtlExitUserProcess()	Unknown
 	kernel32.dll!ExitProcessImplementation()	Unknown
 	ucrtbase.dll!exit_or_terminate_process()	Unknown
 	ucrtbase.dll!common_exit()	Unknown
 	python312.dll!00007ff9cab3ec8d()	Unknown
 	python312.dll!00007ff9cab3efbf()	Unknown
 	python312.dll!00007ff9cab3edee()	Unknown
 	python312.dll!00007ff9cab57f4c()	Unknown
 	python312.dll!00007ff9cab57579()	Unknown
 	python312.dll!00007ff9cab573be()	Unknown
 	python312.dll!00007ff9cab5729b()	Unknown
 	python312.dll!00007ff9cabacfcb()	Unknown
 	python312.dll!00007ff9cabacd7d()	Unknown
 	python312.dll!00007ff9cab99e2d()	Unknown
 	python.exe!00007ff78a641230()	Unknown
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

It might be because the destruct order of some global variables was wrong. I saw DX DLLs were getting destroyed earlier than the WebGPU instance in our code in onnxruntime_pybind11_state.pyd.

@snnn snnn added the ep:WebGPU ort-web webgpu provider label Jan 31, 2025
@snnn snnn marked this pull request as draft January 31, 2025 05:37
@snnn snnn added the merge on green This PR can be merged when the CI is green. Please merge. label Jan 31, 2025
@snnn snnn changed the title Remove Enable pybind from webgpu pipeline Remove "--enable_pybind" from webgpu pipeline Jan 31, 2025
@snnn snnn marked this pull request as ready for review January 31, 2025 07:22
@snnn snnn requested a review from fs-eire January 31, 2025 07:26
@snnn
Copy link
Member Author

snnn commented Jan 31, 2025

/azp run Linux GPU CI Pipeline , iOS CI Pipeline

Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@guschmue
Copy link
Contributor

I have seen this one, happens on linux and macos. need to take a look why this is.

@snnn snnn merged commit 61fae9b into main Jan 31, 2025
118 checks passed
@snnn snnn deleted the snnn/fix_webgpu branch January 31, 2025 16:44
sfatimar pushed a commit to intel/onnxruntime that referenced this pull request Feb 5, 2025
There is a crash in the WebGPU CI pipeline. It crashed at process
shutdown when unloading onnxruntime_pybind11_state.pyd.
Here is the callstack:

```
 	dxil.dll!DxcSwapThreadMalloc()	Unknown
 	dxil.dll!DxcThreadMalloc::DxcThreadMalloc(struct IMalloc *)	Unknown
 	dxil.dll!DxcValidator::Release(void)	Unknown
 	[Inline Frame] webgpu_dawn.dll!Microsoft::WRL::ComPtr<IDxcValidator>::InternalRelease() Line 235	C++
 	[Inline Frame] webgpu_dawn.dll!Microsoft::WRL::ComPtr<IDxcValidator>::{dtor}() Line 290	C++
 	webgpu_dawn.dll!dawn::native::d3d12::Backend::`scalar deleting destructor'(unsigned int)	C++
 	webgpu_dawn.dll!`eh vector destructor iterator'(void * ptr, unsigned __int64 size, unsigned __int64 count, void(*)(void *) destructor)	C++
 	webgpu_dawn.dll!dawn::native::InstanceBase::~InstanceBase() Line 197	C++
 	webgpu_dawn.dll!dawn::native::InstanceBase::`scalar deleting destructor'(unsigned int)	C++
 	webgpu_dawn.dll!dawn::native::InstanceBase::DeleteThis() Line 218	C++
 	ucrtbase.dll!<lambda>(void)()	Unknown
 	ucrtbase.dll!__crt_seh_guarded_call<int>::operator()<<lambda_7777bce6b2f8c936911f934f8298dc43>,<lambda>(void) &,<lambda_3883c3dff614d5e0c5f61bb1ac94921c>>()	Unknown
 	ucrtbase.dll!_execute_onexit_table()	Unknown
 	onnxruntime_pybind11_state.pyd!dllmain_crt_process_detach(const bool is_terminating) Line 182	C++
>	onnxruntime_pybind11_state.pyd!dllmain_dispatch(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 293	C++
 	ntdll.dll!LdrpCallInitRoutine()	Unknown
 	ntdll.dll!LdrShutdownProcess()	Unknown
 	ntdll.dll!RtlExitUserProcess()	Unknown
 	kernel32.dll!ExitProcessImplementation()	Unknown
 	ucrtbase.dll!exit_or_terminate_process()	Unknown
 	ucrtbase.dll!common_exit()	Unknown
 	python312.dll!00007ff9cab3ec8d()	Unknown
 	python312.dll!00007ff9cab3efbf()	Unknown
 	python312.dll!00007ff9cab3edee()	Unknown
 	python312.dll!00007ff9cab57f4c()	Unknown
 	python312.dll!00007ff9cab57579()	Unknown
 	python312.dll!00007ff9cab573be()	Unknown
 	python312.dll!00007ff9cab5729b()	Unknown
 	python312.dll!00007ff9cabacfcb()	Unknown
 	python312.dll!00007ff9cabacd7d()	Unknown
 	python312.dll!00007ff9cab99e2d()	Unknown
 	python.exe!00007ff78a641230()	Unknown
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown
```
It might be because the destruct order of some global variables was
wrong. I saw DX DLLs were getting destroyed earlier than the WebGPU
instance in our code in onnxruntime_pybind11_state.pyd.
sfatimar pushed a commit to intel/onnxruntime that referenced this pull request Feb 5, 2025
There is a crash in the WebGPU CI pipeline. It crashed at process
shutdown when unloading onnxruntime_pybind11_state.pyd.
Here is the callstack:

```
 	dxil.dll!DxcSwapThreadMalloc()	Unknown
 	dxil.dll!DxcThreadMalloc::DxcThreadMalloc(struct IMalloc *)	Unknown
 	dxil.dll!DxcValidator::Release(void)	Unknown
 	[Inline Frame] webgpu_dawn.dll!Microsoft::WRL::ComPtr<IDxcValidator>::InternalRelease() Line 235	C++
 	[Inline Frame] webgpu_dawn.dll!Microsoft::WRL::ComPtr<IDxcValidator>::{dtor}() Line 290	C++
 	webgpu_dawn.dll!dawn::native::d3d12::Backend::`scalar deleting destructor'(unsigned int)	C++
 	webgpu_dawn.dll!`eh vector destructor iterator'(void * ptr, unsigned __int64 size, unsigned __int64 count, void(*)(void *) destructor)	C++
 	webgpu_dawn.dll!dawn::native::InstanceBase::~InstanceBase() Line 197	C++
 	webgpu_dawn.dll!dawn::native::InstanceBase::`scalar deleting destructor'(unsigned int)	C++
 	webgpu_dawn.dll!dawn::native::InstanceBase::DeleteThis() Line 218	C++
 	ucrtbase.dll!<lambda>(void)()	Unknown
 	ucrtbase.dll!__crt_seh_guarded_call<int>::operator()<<lambda_7777bce6b2f8c936911f934f8298dc43>,<lambda>(void) &,<lambda_3883c3dff614d5e0c5f61bb1ac94921c>>()	Unknown
 	ucrtbase.dll!_execute_onexit_table()	Unknown
 	onnxruntime_pybind11_state.pyd!dllmain_crt_process_detach(const bool is_terminating) Line 182	C++
>	onnxruntime_pybind11_state.pyd!dllmain_dispatch(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 293	C++
 	ntdll.dll!LdrpCallInitRoutine()	Unknown
 	ntdll.dll!LdrShutdownProcess()	Unknown
 	ntdll.dll!RtlExitUserProcess()	Unknown
 	kernel32.dll!ExitProcessImplementation()	Unknown
 	ucrtbase.dll!exit_or_terminate_process()	Unknown
 	ucrtbase.dll!common_exit()	Unknown
 	python312.dll!00007ff9cab3ec8d()	Unknown
 	python312.dll!00007ff9cab3efbf()	Unknown
 	python312.dll!00007ff9cab3edee()	Unknown
 	python312.dll!00007ff9cab57f4c()	Unknown
 	python312.dll!00007ff9cab57579()	Unknown
 	python312.dll!00007ff9cab573be()	Unknown
 	python312.dll!00007ff9cab5729b()	Unknown
 	python312.dll!00007ff9cabacfcb()	Unknown
 	python312.dll!00007ff9cabacd7d()	Unknown
 	python312.dll!00007ff9cab99e2d()	Unknown
 	python.exe!00007ff78a641230()	Unknown
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown
```
It might be because the destruct order of some global variables was
wrong. I saw DX DLLs were getting destroyed earlier than the WebGPU
instance in our code in onnxruntime_pybind11_state.pyd.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ep:WebGPU ort-web webgpu provider merge on green This PR can be merged when the CI is green. Please merge.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants