Enable SHORTFIN_ENABLE_TOKENIZERS in Linux package builds

[ScottTodd]

This will allow us to replace the current loosely coupled tokenizers Python package (https://pypi.org/project/tokenizers/) with a source dependency on https://github.com/mlc-ai/tokenizers-cpp that provides bindings to the underlying rust library from https://github.com/huggingface/tokenizers and C++ library from https://github.com/google/sentencepiece.

See also:

• libshortfin CI and Releases #130
• [shortfin] Add C++ tokenizer wrapper library. #610
  • Note the future work items listed there
  • We'll still need to plumb through a Python API and update usage in files like https://github.com/nod-ai/shark-ai/blob/main/shortfin/python/shortfin_apps/llm/components/tokenizer.py
• Enable building shortfin's Rust dependencies on Windows #620

Main tasks for this issue:

• Add rust/cargo to a dockerfile in https://github.com/nod-ai/base-docker-images/
• Switch this script to use that dockerfile:

  shark-ai/shortfin/build_tools/build_linux_package.sh

  Lines 40 to 41 in 1e26b20

  	# Note: we can switch to https://github.com/nod-ai/base-docker-images as needed for extra deps.
  	MANYLINUX_DOCKER_IMAGE="${MANYLINUX_DOCKER_IMAGE:-quay.io/pypa/manylinux_2_28_${ARCH}:latest}"

• Set SHORTFIN_ENABLE_TOKENIZERS in that script
• Ensure that https://github.com/nod-ai/shark-ai/blob/main/.github/workflows/build_packages.yml picks up the changes

[ScottTodd]

We also can't use a github action to set up rust on the host machine like the shortfin CI workflow:

shark-ai/.github/workflows/ci-libshortfin.yml

Lines 91 to 97 in 7e62c25

	- name: Setup Rust
	# For now, `SHORTFIN_ENABLE_TOKENIZERS` is only enabled for 'Ubuntu (Clang)(full)'.
	# TODO(#620): Enable on Windows.
	if: ${{ matrix.name == 'Ubuntu (Clang)(full)'}}
	uses: dtolnay/rust-toolchain@315e265cd78dad1e1dcf3a5074f6d6c47029d5aa # master branch (Nov 18, 2024)
	with:
	toolchain: stable

since the Linux package builds are run in a manylinux docker container:

shark-ai/shortfin/build_tools/build_linux_package.sh

Lines 175 to 180 in 7e62c25

	# Trampoline to the docker container if running on the host.
	if [ -z "${__MANYLINUX_BUILD_WHEELS_IN_DOCKER-}" ]; then
	run_on_host "$@"
	else
	run_in_docker "$@"
	fi