Total number of attention heads (X) must be divisible by tensor parallel size (Y).

[tamastarjanyi]

Trying to run falcon-7b on multiple nodes however getting the below error. Which is funny since 71 is a prime number. So I can run it on either 1 GPU (1NODE) or on 71 GPUs (NODES). Is there any way to avoid this problem?

My config is

RayHead is running on one node (actually container within kubernetes) via
ray start --head --dashboard-host 0.0.0.0 --num-gpus 1 --num-cpus 7

And RayWorker is working in another container via
ray start --disable-usage-stats --num-gpus 1 --num-cpus 7 --address <address>
ray status is fine

Usage:
 0.0/14.0 CPU
 0.0/2.0 GPU (0.0 used of 2.0 reserved in placement groups)
 0B/1.94TiB memory
 0B/18.63GiB object_store_memory

But when trying to run falcon-7b via
python -m vllm.entrypoints.api_server --model tiiuae/falcon-7b --trust-remote-code --tensor-parallel-size 2 --port 8080 --engine-use-ray --worker-use-ray

Below error is raised
Total number of attention heads (71) must be divisible by tensor parallel size (2).

[oushu1zhangxiangxuan1]

Same problem with me: --tensor-parallel-size

1. SET to 6: Total number of attention heads (40) must be divisible by tensor parallel size (6).
2. SET to 5:

2024-01-22 14:32:19,671 INFO worker.py:1540 -- Connecting to existing Ray cluster at address: 
2024-01-22 14:32:19,679 INFO worker.py:1724 -- Connected to Ray cluster.
INFO 01-22 14:32:19 llm_engine.py:73] Initializing an LLM engine with config: model='/tmp/CodeLlama-13b-Instruct-hf', tokenizer='/tmp/CodeLlama-13b-Instruct-hf', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=5, quantization=None, enforce_eager=False, seed=0)
2024-01-22 14:32:27 | ERROR | stderr | Traceback (most recent call last):
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/runpy.py", line 196, in _run_module_as_main
2024-01-22 14:32:27 | ERROR | stderr |     return _run_code(code, main_globals, None,
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/runpy.py", line 86, in _run_code
2024-01-22 14:32:27 | ERROR | stderr |     exec(code, run_globals)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/fastchat/serve/vllm_worker.py", line 259, in <module>
2024-01-22 14:32:27 | ERROR | stderr |     engine = AsyncLLMEngine.from_engine_args(engine_args)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 496, in from_engine_args
2024-01-22 14:32:27 | ERROR | stderr |     engine = cls(parallel_config.worker_use_ray,
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 269, in __init__
2024-01-22 14:32:27 | ERROR | stderr |     self.engine = self._init_engine(*args, **kwargs)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 314, in _init_engine
2024-01-22 14:32:27 | ERROR | stderr |     return engine_class(*args, **kwargs)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 108, in __init__
2024-01-22 14:32:27 | ERROR | stderr |     self._init_workers_ray(placement_group)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 195, in _init_workers_ray
2024-01-22 14:32:27 | ERROR | stderr |     self._run_workers(
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 755, in _run_workers
2024-01-22 14:32:27 | ERROR | stderr |     self._run_workers_in_batch(workers, method, *args, **kwargs))
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 732, in _run_workers_in_batch
2024-01-22 14:32:27 | ERROR | stderr |     all_outputs = ray.get(all_outputs)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
2024-01-22 14:32:27 | ERROR | stderr |     return fn(*args, **kwargs)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
2024-01-22 14:32:27 | ERROR | stderr |     return func(*args, **kwargs)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/ray/_private/worker.py", line 2624, in get
2024-01-22 14:32:27 | ERROR | stderr |     raise value.as_instanceof_cause()
2024-01-22 14:32:27 | ERROR | stderr | ray.exceptions.RayTaskError(AssertionError): ray::RayWorkerVllm.execute_method() (pid=9107, ip=172.21.32.7, actor_id=5e6bd6188901024c707bb3ba07000000, repr=<vllm.engine.ray_utils.RayWorkerVllm object at 0x7f7c1d248b50>)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/engine/ray_utils.py", line 31, in execute_method
2024-01-22 14:32:27 | ERROR | stderr |     return executor(*args, **kwargs)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/worker/worker.py", line 79, in load_model
2024-01-22 14:32:27 | ERROR | stderr |     self.model_runner.load_model()
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 57, in load_model
2024-01-22 14:32:27 | ERROR | stderr |     self.model = get_model(self.model_config)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/model_executor/model_loader.py", line 65, in get_model
2024-01-22 14:32:27 | ERROR | stderr |     model = model_class(model_config.hf_config, linear_method)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 275, in __init__
2024-01-22 14:32:27 | ERROR | stderr |     self.model = LlamaModel(config, linear_method)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 233, in __init__
2024-01-22 14:32:27 | ERROR | stderr |     self.embed_tokens = VocabParallelEmbedding(
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/model_executor/layers/vocab_parallel_embedding.py", line 63, in __init__
2024-01-22 14:32:27 | ERROR | stderr |     vocab_range_from_global_vocab_size(
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/model_executor/layers/vocab_parallel_embedding.py", line 31, in vocab_range_from_global_vocab_size
2024-01-22 14:32:27 | ERROR | stderr |     per_partition_vocab_size = divide(global_vocab_size, world_size)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/model_executor/parallel_utils/utils.py", line 19, in divide
2024-01-22 14:32:27 | ERROR | stderr |     ensure_divisibility(numerator, denominator)
2024-01-22 14:32:27 | ERROR | stderr |   File "/root/space/conda_envs/t2c12/lib/python3.10/site-packages/vllm/model_executor/parallel_utils/utils.py", line 12, in ensure_divisibility
2024-01-22 14:32:27 | ERROR | stderr |     assert numerator % denominator == 0, "{} is not divisible by {}".format(
2024-01-22 14:32:27 | ERROR | stderr | AssertionError: 32064 is not divisible by 5

[eigen2017]

same error，32 heads on 3 gpus

[Luowaterbi]

same error, I use starcoder2 but tells me ValueError: Total number of attention heads (36) must be divisible by tensor parallel size (8).

[hubatish]

What controls total number of attention heads? Can I decrease / change that number rather than change the number of gpus? It doesn't seem to be an available vllm arg:
https://docs.vllm.ai/en/latest/serving/engine_args.html

[SmartManoj]

#5500 (comment)