[Ascend] Enable support for 310p NPUs

[zer0py2c]

Adapted models

• ChatGLM3-6B
• vicuna-7b-v1.3

⚠️Before loading ChatGLM3-6B, you need to modify the model code file!

• modeling_chatglm.py

...
# @torch.jit.script
def apply_rotary_pos_emb(x: torch.Tensor, rope_cache: torch.Tensor) -> torch.Tensor:
    ...

Environment Preparation

ℹ️Recommend that build a Docker image for environment setup.

• Docker: 20.10.2+

  • Ascend Docker Runtime Plugin: Download

• NPU Driver and Firmware: Download Page

• CANN: 8.0.RC2

• Python: 3.8+

  • torch-npu==2.1.0.post6
  • transformers>=4.41.0

Inference with Command Line Interface

Single NPU

python3 -m fastchat.serve.cli --model-path /opt/models/vicuna-7b-v1.3/ --device npu

Multiple NPUs

ASCEND_RT_VISIBLE_DEVICES=0,1 python3 -m fastchat.serve.cli --model-path /opt/models/vicuna-7b-v1.3/ --num-gpus 2 --device npu --max-gpu-memory 12GiB