Quantization Configuration in VLM Model

[aptsunny]

I've been going through the latest commit with ID 69d7dc0 and noticed that there seems to be an issue with the quantization configuration for the VLM model. The original configuration file is configs/quantization/methods/Awq/awq_w_only_vlm.yml, which is shown below:

calib:
    name: custom_mm
    download: False
    path: calib data path
    apply_chat_template: True
    add_answer: True # Default is False. If set it to True, calib data will add answers.
    n_samples: 8
    bs: -1
    seq_len: 512
    padding: True
    seed: *seed

I've found that this configuration for the calibration dataset can lead to a mismatch in the model dimensions for the LLM part. This issue has been observed in both InternVL2 and Qwen2VL models. I've attached the bug log files for your reference.
I believe resolving this would greatly improve the performance and stability of the VLM model. If you need any further information or clarification for bug verification, please feel free to reply to this issue on GitHub. Thank you so much for your time and effort.
Qwen2VL_awq.log

InternVL2_awq.log

[XiaotaoChen]

Hi, i met another bug when quantizing internVL2 ref the config: configs/quantization/methods/Awq/awq_w_only_vlm.yml, the detail info is here #303. have you met it?

[aptsunny]

Hi, i met another bug when quantizing internVL2 ref the config: configs/quantization/methods/Awq/awq_w_only_vlm.yml, the detail info is here #303. have you met it?

I recommend checking the download method for the MME dataset carefully

[XiaotaoChen]

Hi, i met another bug when quantizing internVL2 ref the config: configs/quantization/methods/Awq/awq_w_only_vlm.yml, the detail info is here #303. have you met it?

I recommend checking the download method for the MME dataset carefully

thanks. it's solved