[Bug] Qwen2-VL-7B with sglang has significant numerical calculation errors compared to HF Transformers

[kritohyh]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

In practice, we found that sglang Qwen2-VL model has numerical calculation errors compared to HF Transformers model in both Qwen2VisionTransformer and Qwen2Model parts.
Our input image has 720 tokens input to Vit encoding, and the lowest embedded cosine similarity in the output is 0.1775. In addition, we directly feed the Vit output and text embedding of Transformers to the LLM part. The cosine similarity of the lowest among the 830 HS outputs in the Prefill stage is reduced to 0.9499, and in the generation stage, as the output increases, the cosine similarity may further decrease to 0.580.

Here are the code blocks we found that caused the differences：
RMSNorm's CUDA Kernel and PyTorch Native
Residual Sum Precision in Transformer Blocks
QKVParallelLinear, RowParallelLinear vs nn.Linear in Transformer Blocks
Merge Column Parallel Linear, Row Parallel Linear vs nn.Linear in Qwen2MLP
Calculation of sin/cos cache in RotaryEmbedding

After eliminating the above differences, we achieved precision alignment. But for performance reasons, is there a repair plan for the above issues？

Reproduction

Qwen2-VL

Environment

Python: 3.10.16 (main, Dec 4 2024, 08:53:37) [GCC 9.4.0]
CUDA available: True
GPU 0: NVIDIA GeForce RTX 4090 D
GPU 0 Compute Capability: 8.9
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.1, V12.1.105
CUDA Driver Version: 550.90.12
PyTorch: 2.5.1+cu124
sglang: 0.4.0.post1
flashinfer: 0.1.6
triton: 3.1.0
transformers: 4.45.2
torchao: 0.7.0
numpy: 1.26.4
aiohttp: 3.11.11
fastapi: 0.115.6
hf_transfer: 0.1.8
huggingface_hub: 0.27.0
interegular: 0.3.3
modelscope: 1.18.1
orjson: 3.10.12
packaging: 24.2
psutil: 6.1.1
pydantic: 2.10.4
multipart: 0.0.20
zmq: 26.2.0
uvicorn: 0.34.0
uvloop: 0.21.0
vllm: 0.6.4.post1
openai: 1.58.1
anthropic: 0.42.0
decord: 0.6.0
NVIDIA Topology:
GPU0 NIC0 NIC1 NIC2 NIC3 NIC4 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X SYS SYS SYS SYS SYS N/A
NIC0 SYS X PHB PHB PHB PHB
NIC1 SYS PHB X PHB PHB PHB
NIC2 SYS PHB PHB X PHB PHB
NIC3 SYS PHB PHB PHB X PHB
NIC4 SYS PHB PHB PHB PHB X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4

Hypervisor vendor: KVM
ulimit soft: 1048576

[zhaochenyang20]

Thanks. We have noticed this for long. This is urgent to us.

[yizhang2077]

FYI, The vit encoding accuracy problem may be alleviated once #2977 has been merged, but we still need do some tests.