diff --git a/examples/st_covost2/README.md b/examples/st_covost2/README.md index 13d708f..2864311 100755 --- a/examples/st_covost2/README.md +++ b/examples/st_covost2/README.md @@ -1,5 +1,15 @@ # ST_covost2 + +## Model Stracture +示例图片 + + +## Multitask +示例图片 + + + ## Download Model We only train the q-former projector in this recipe. Encoder | Projector | LLM @@ -33,7 +43,7 @@ You can find the test jsonl in "test_st.jsonl" {"audio": "/userhome/speech/data/common/4/en/clips/common_voice_en_699711.mp3", "prompt": "\"She'll be all right.\"<|zh|>", "gt": "\"She'll be all right.\"<|zh|>她会没事的。", "source": "covost_enenzh"} ``` ## Train Stage -Here, we have designed a four-step training process, where each training session uses the checkpoint obtained from the previous training session. +Here, we have designed a three-step training process, where each training session uses the checkpoint obtained from the previous training session. ``` #In this step, we perform ASR pretraining to acquire speech recognition capabilities. bash asr_pretrain.sh @@ -41,10 +51,8 @@ bash asr_pretrain.sh #In this phase, we conduct multimodal machine translation training to enhance the final performance. bash mmt.sh -#monolingual SRT training. +#monolingual SRT training and multitask training. bash srt.sh - -#multilingual multitask training. bash zsrt.sh ``` @@ -53,7 +61,7 @@ bash zsrt.sh You can try our pre-trained model. ``` -bash infer.sh +bash infer_enzh.sh ``` ## Citation diff --git a/examples/st_covost2/image/framework.jpg b/examples/st_covost2/image/framework.jpg new file mode 100644 index 0000000..d0f746e Binary files /dev/null and b/examples/st_covost2/image/framework.jpg differ diff --git a/examples/st_covost2/image/prompt.png b/examples/st_covost2/image/prompt.png new file mode 100644 index 0000000..4880749 Binary files /dev/null and b/examples/st_covost2/image/prompt.png differ diff --git a/examples/st_covost2/scripts/infer.sh b/examples/st_covost2/scripts/infer_enzh.sh similarity index 100% rename from examples/st_covost2/scripts/infer.sh rename to examples/st_covost2/scripts/infer_enzh.sh