Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable building shortfin's Rust dependencies on Windows #620

Open
marbre opened this issue Nov 27, 2024 · 0 comments
Open

Enable building shortfin's Rust dependencies on Windows #620

marbre opened this issue Nov 27, 2024 · 0 comments
Labels
infra General category for infrastructure-related requests for common triaging and prioritization

Comments

@marbre
Copy link
Collaborator

marbre commented Nov 27, 2024

When using ilammy/msvc-dev-cmd to set up MSVC and dtolnay/rust-toolchain to set up Rust, building with SHORTFIN_ENABLE_TOKENIZERS set to ON fails on Windows with:

error: linking with `link.exe` failed: exit code: 1
  |
  = note: "C:\\Program Files\\Git\\usr\\bin\\link.exe" "/NOLOGO" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcx6IhZz\\symbols.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\crossbeam-utils-476df0f36b4e4560\\build_script_build-476df0f36b4e4560.build_script_build.81593264f59a8625-cgu.0.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\crossbeam-utils-476df0f36b4e4560\\build_script_build-476df0f36b4e4560.build_script_build.81593264f59a8625-cgu.1.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\crossbeam-utils-476df0f36b4e4560\\build_script_build-476df0f36b4e4560.build_script_build.81593264f59a8625-cgu.2.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\crossbeam-utils-476df0f36b4e4560\\build_script_build-476df0f36b4e4560.30tdflta5f29rs5gf55oknqee.rcgu.o" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd-2df1f22abef96888.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libpanic_unwind-7fa781213a0698f8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libwindows_targets-2440cb72ce7deb9b.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_demangle-f04b9120076f20fa.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd_detect-b521ee511095af2f.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libhashbrown-124aa6c4c6ef4b4c.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_alloc-c86a42f7194744c8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunwind-a4160695964[735](https://github.com/nod-ai/shark-ai/actions/runs/12051210540/job/33601745584?pr=619#step:10:736)08.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcfg_if-e246a9218bd1ed0e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc-8f9b5fcbcd27c22e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_core-65178e86c6c71ba8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcore-fbeb171b69c59b37.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-e3a3e7896142045d.rlib" "kernel32.lib" "kernel32.lib" "advapi32.lib" "ntdll.lib" "userenv.lib" "ws2_32.lib" "dbghelp.lib" "/defaultlib:msvcrt" "/NXCOMPAT" "/OUT:D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\crossbeam-utils-476df0f36b4e4560\\build_script_build-476df0f36b4e4560.exe" "/OPT:REF,NOICF" "/DEBUG" "/PDBALTPATH:%_PDB%" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libcore.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libstd.natvis"
  = note: /usr/bin/link: extra operand 'D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\crossbeam-utils-476df0f36b4e4560\\build_script_build-476df0f36b4e4560.build_script_build.81593264f59a8625-cgu.0.rcgu.o'
          Try '/usr/bin/link --help' for more information.
          

error: linking with `link.exe` failed: exit code: 1
  |
  = note: "C:\\Program Files\\Git\\usr\\bin\\link.exe" "/NOLOGO" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustc9oR3ZA\\symbols.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\proc-macro2-aa639117d62021a0\\build_script_build-aa639117d62021a0.build_script_build.2e37346e7cb754b8-cgu.0.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\proc-macro2-aa639117d62021a0\\build_script_build-aa639117d62021a0.build_script_build.2e37346e7cb754b8-cgu.1.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\proc-macro2-aa639117d62021a0\\build_script_build-aa639117d62021a0.eo77g25asojiq7m7449m2h1zo.rcgu.o" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd-2df1f22abef96888.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libpanic_unwind-7fa781213a0698f8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libwindows_targets-2440cb72ce7deb9b.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_demangle-f04b9120076f20fa.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd_detect-b521ee511095af2f.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libhashbrown-124aa6c4c6ef4b4c.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_alloc-c86a42f7194744c8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunwind-a416069596473508.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcfg_if-e246a9218bd1ed0e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc-8f9b5fcbcd27c22e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_core-65178e86c6c71ba8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcore-fbeb171b69c59b37.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-e3a3e7896142045d.rlib" "kernel32.lib" "kernel32.lib" "advapi32.lib" "ntdll.lib" "userenv.lib" "ws2_32.lib" "dbghelp.lib" "/defaultlib:msvcrt" "/NXCOMPAT" "/OUT:D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\proc-macro2-aa639117d62021a0\\build_script_build-aa639117d62021a0.exe" "/OPT:REF,NOICF" "/DEBUG" "/PDBALTPATH:%_PDB%" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libcore.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libstd.natvis"
  = note: /usr/bin/link: extra operand 'D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\proc-macro2-aa639117d62021a0\\build_script_build-aa639117d62021a0.build_script_build.2e37346e7cb754b8-cgu.0.rcgu.o'
          Try '/usr/bin/link --help' for more information.
          

note: `link.exe` returned an unexpected error

note: the Visual Studio build tools may need to be repaired using the Visual Studio installer

note: or a necessary component may be missing from the "C++ build tools" workload

   Compiling shlex v1.3.0
   Compiling serde v1.0.215
error: could not compile `proc-macro2` (build script) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
error: could not compile `crossbeam-utils` (build script) due to 1 previous error
error: linking with `link.exe` failed: exit code: 1
  |
  = note: "C:\\Program Files\\Git\\usr\\bin\\link.exe" "/NOLOGO" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcKHwh9K\\symbols.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\serde-9df6f3f78162638f\\build_script_build-9df6f3f78162638f.build_script_build.678f91059671b4c4-cgu.0.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\serde-9df6f3f78162638f\\build_script_build-9df6f3f78162638f.build_script_build.678f91059671b4c4-cgu.1.rcgu.o" "D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\serde-9df6f3f78162638f\\build_script_build-9df6f3f78162638f.bfk8wi8a449l6t5jxbcehj784.rcgu.o" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd-2df1f22abef96888.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libpanic_unwind-7fa781213a0698f8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libwindows_targets-2440cb72ce7deb9b.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_demangle-f04b9120076f20fa.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd_detect-b521ee511095af2f.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libhashbrown-124aa6c4c6ef4b4c.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_alloc-c86a42f7194[744](https://github.com/nod-ai/shark-ai/actions/runs/12051210540/job/33601745584?pr=619#step:10:745)c8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunwind-a416069596473508.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcfg_if-e246a9218bd1ed0e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc-8f9b5fcbcd27c22e.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_core-65178e86c6c71ba8.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcore-fbeb171b69c59b37.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-e3a3e7896142045d.rlib" "kernel32.lib" "kernel32.lib" "advapi32.lib" "ntdll.lib" "userenv.lib" "ws2_32.lib" "dbghelp.lib" "/defaultlib:msvcrt" "/NXCOMPAT" "/OUT:D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\serde-9df6f3f78162638f\\build_script_build-9df6f3f78162638f.exe" "/OPT:REF,NOICF" "/DEBUG" "/PDBALTPATH:%_PDB%" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libcore.natvis" "/NATVIS:C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libstd.natvis"
  = note: /usr/bin/link: extra operand 'D:/a/shark-ai/shark-ai/shortfin/build/_deps/tokenizers_cpp-build\\debug\\build\\serde-9df6f3f78162638f\\build_script_build-9df6f3f78162638f.build_script_build.678f91059671b4c4-cgu.0.rcgu.o'
          Try '/usr/bin/link --help' for more information.
          

error: could not compile `serde` (build script) due to 1 previous error
ninja: build stopped: subcommand failed.

This is not necessarily related to using the actiondtolnay/rust-toolchain and is probably rather a problem with ilammy/msvc-dev-cmd + Rust. GitHub rusted runners come with rustup preinstalled but setting up a toolchain with rustup default stable instead of using dtolnay/rust-toolchain results in similar build failures. If no MSVC is set up before, rustup-init offers to take care and automatically install the prerequisites, see https://rust-lang.github.io/rustup/installation/windows-msvc.html, which is needs a closer look.

marbre added a commit to marbre/shark-ai that referenced this issue Nov 27, 2024
Sets up Rust to build shortfin with `SHORTFIN_ENABLE_TOKENIZERS` set to
`ON` on Linux. At the moment, building with Rust is limited to Linux as
there are build failures on Windows (see issue nod-ai#620).
@marbre marbre added the infra General category for infrastructure-related requests for common triaging and prioritization label Nov 27, 2024
marbre added a commit to marbre/shark-ai that referenced this issue Dec 9, 2024
Sets up Rust to build shortfin with `SHORTFIN_ENABLE_TOKENIZERS` set to
`ON` on Linux. At the moment, building with Rust is limited to Linux as
there are build failures on Windows (see issue nod-ai#620).
marbre added a commit that referenced this issue Dec 9, 2024
Sets up Rust to build shortfin with `SHORTFIN_ENABLE_TOKENIZERS` set to
`ON` on Linux. At the moment, building with Rust is limited to Linux as
there are build failures on Windows (see issue #620).
IanNod pushed a commit to IanNod/SHARK-Platform that referenced this issue Dec 17, 2024
Sets up Rust to build shortfin with `SHORTFIN_ENABLE_TOKENIZERS` set to
`ON` on Linux. At the moment, building with Rust is limited to Linux as
there are build failures on Windows (see issue nod-ai#620).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infra General category for infrastructure-related requests for common triaging and prioritization
Projects
None yet
Development

No branches or pull requests

1 participant