Clarifying how to calculate the KV cache usage in GiB

[googlercolin]

I would like to calculate the usage of the KV cache in GiB, however, I am not very sure if num_total_gpu = self.cache_config.num_gpu_blocks in the code means the total number of blocks allocated for the KV cache (6.38GiB as per my logging output below) or the total number of GPU blocks (39.50GiB)?

Also, when scheduler.block_manager.get_num_free_gpu_blocks() is called, are the free blocks only taken from those allocated to for the KV cache (part of the 6.38GiB), or it includes the 0.10 (3.95 GiB) outside the gpu_memory_utilization?

Logs:

INFO 02-24 07:59:56 model_runner.py:1115] Loading model weights took 27.5915 GB
INFO 02-24 07:59:58 worker.py:266] Memory profiling takes 1.82 seconds
INFO 02-24 07:59:58 worker.py:266] the current vLLM instance can use total_gpu_memory (39.50GiB) x gpu_memory_utilization (0.90) = 35.55GiB
INFO 02-24 07:59:58 worker.py:266] model weights take 27.59GiB; non_torch_memory takes 0.09GiB; PyTorch activation peak memory takes 1.48GiB; the rest of the memory reserved for KV Cache is 6.38GiB.

Thank you!