Skip to content
Back to pull request #12047

fix the issue during batched inference of Sortformer diarizer #8169

Sign in to view logs

fix the issue during batched inference of Sortformer diarizer

fix the issue during batched inference of Sortformer diarizer #8169

Triggered via pull request February 7, 2025 21:21
@tango4jtango4j
labeled #12047
fix/sortformer_batch_len
Status Failure
Total duration 8h 6m 27s
Artifacts 22

cicd-main.yml

on: pull_request
pre-flight
7s
pre-flight
cicd-test-container-build  /  main
4m 45s
cicd-test-container-build / main
L0_Unit_Tests_GPU_ASR  /  main
13m 6s
L0_Unit_Tests_GPU_ASR / main
L0_Unit_Tests_GPU_Audio  /  main
2m 21s
L0_Unit_Tests_GPU_Audio / main
L0_Unit_Tests_GPU_Common  /  main
4m 14s
L0_Unit_Tests_GPU_Common / main
L0_Unit_Tests_GPU_LLM  /  main
2m 8s
L0_Unit_Tests_GPU_LLM / main
L0_Unit_Tests_GPU_Multimodal  /  main
1m 54s
L0_Unit_Tests_GPU_Multimodal / main
L0_Unit_Tests_GPU_NLP  /  main
4m 17s
L0_Unit_Tests_GPU_NLP / main
L0_Unit_Tests_GPU_TTS  /  main
1m 45s
L0_Unit_Tests_GPU_TTS / main
L0_Unit_Tests_GPU_Hydra  /  main
2m 25s
L0_Unit_Tests_GPU_Hydra / main
L0_Unit_Tests_GPU_Lightning  /  main
4m 28s
L0_Unit_Tests_GPU_Lightning / main
L0_Unit_Tests_GPU_Others  /  main
3m 14s
L0_Unit_Tests_GPU_Others / main
L0_Unit_Tests_CPU_ASR  /  main
10m 6s
L0_Unit_Tests_CPU_ASR / main
L0_Unit_Tests_CPU_Audio  /  main
1m 49s
L0_Unit_Tests_CPU_Audio / main
L0_Unit_Tests_CPU_Common  /  main
3m 15s
L0_Unit_Tests_CPU_Common / main
L0_Unit_Tests_CPU_LLM  /  main
1m 39s
L0_Unit_Tests_CPU_LLM / main
L0_Unit_Tests_CPU_Multimodal  /  main
1m 37s
L0_Unit_Tests_CPU_Multimodal / main
L0_Unit_Tests_CPU_NLP  /  main
2m 14s
L0_Unit_Tests_CPU_NLP / main
L0_Unit_Tests_CPU_TTS  /  main
2m 40s
L0_Unit_Tests_CPU_TTS / main
L0_Unit_Tests_CPU_Core  /  main
4m 17s
L0_Unit_Tests_CPU_Core / main
L0_Unit_Tests_CPU_Hydra  /  main
1m 58s
L0_Unit_Tests_CPU_Hydra / main
L0_Unit_Tests_CPU_Lightning  /  main
1m 48s
L0_Unit_Tests_CPU_Lightning / main
L0_Unit_Tests_CPU_Others  /  main
2m 15s
L0_Unit_Tests_CPU_Others / main
cicd-import-tests
1m 27s
cicd-import-tests
L0_Unit_Tests_GPU_Core  /  main
3m 20s
L0_Unit_Tests_GPU_Core / main
L2_Community_LLM_Checkpoints_tests_Bert  /  main
2m 2s
L2_Community_LLM_Checkpoints_tests_Bert / main
L2_Community_LLM_Checkpoints_tests_Mamba2  /  main
1m 54s
L2_Community_LLM_Checkpoints_tests_Mamba2 / main
L2_Community_LLM_Checkpoints_tests_Llama  /  main
49s
L2_Community_LLM_Checkpoints_tests_Llama / main
L2_Community_LLM_Checkpoints_tests_Llama3  /  main
50s
L2_Community_LLM_Checkpoints_tests_Llama3 / main
L2_Community_LLM_Checkpoints_tests_StarCoder  /  main
1m 42s
L2_Community_LLM_Checkpoints_tests_StarCoder / main
L2_Community_LLM_Checkpoints_tests_Falcon  /  main
1m 50s
L2_Community_LLM_Checkpoints_tests_Falcon / main
L2_Community_vita_Checkpoints_tests_Llama3  /  main
5m 2s
L2_Community_vita_Checkpoints_tests_Llama3 / main
L2_PTQ_Llama2_FP8  /  main
2m 44s
L2_PTQ_Llama2_FP8 / main
L2_Distill_Llama2  /  main
3m 0s
L2_Distill_Llama2 / main
L2_Prune_Width_Llama2  /  main
2m 24s
L2_Prune_Width_Llama2 / main
L2_Prune_Depth_Llama2  /  main
2m 12s
L2_Prune_Depth_Llama2 / main
ASR_dev_run_Speech_to_Text  /  main
50s
ASR_dev_run_Speech_to_Text / main
ASR_dev_run_Speech_to_Text_WPE_-_CitriNet  /  main
50s
ASR_dev_run_Speech_to_Text_WPE_-_CitriNet / main
ASR_dev_run_Speech_Pre-training_-_CitriNet  /  main
41s
ASR_dev_run_Speech_Pre-training_-_CitriNet / main
ASR_dev_run_Speech_To_Text_Finetuning  /  main
1m 0s
ASR_dev_run_Speech_To_Text_Finetuning / main
ASR_dev_run_Speech_To_Text_HF_Finetuning  /  main
1m 53s
ASR_dev_run_Speech_To_Text_HF_Finetuning / main
ASR_dev_run_Speech_to_Text_WPE_-_Conformer  /  main
40s
ASR_dev_run_Speech_to_Text_WPE_-_Conformer / main
ASR_dev_run-part_two_Speech_to_Text_WPE_-_Squeezeformer  /  main
36s
ASR_dev_run-part_two_Speech_to_Text_WPE_-_Squeezeformer / main
L2_Speech_to_Text_EMA  /  main
1m 53s
L2_Speech_to_Text_EMA / main
L2_Speech_to_Text_AED  /  main
1m 38s
L2_Speech_to_Text_AED / main
L2_Speaker_dev_run_Speaker_Recognition  /  main
36s
L2_Speaker_dev_run_Speaker_Recognition / main
L2_Speaker_dev_run_Speaker_Diarization  /  main
37s
L2_Speaker_dev_run_Speaker_Diarization / main
L2_Speaker_dev_run_EndtoEnd_Speaker_Diarization_Sortformer  /  main
38s
L2_Speaker_dev_run_EndtoEnd_Speaker_Diarization_Sortformer / main
L2_Speaker_dev_run_EndtoEnd_Diarizer_Inference  /  main
1m 25s
L2_Speaker_dev_run_EndtoEnd_Diarizer_Inference / main
L2_Speaker_dev_run_Speech_to_Label  /  main
40s
L2_Speaker_dev_run_Speech_to_Label / main
L2_Speaker_dev_run_Speaker_Diarization_with_ASR_Inference  /  main
1m 38s
L2_Speaker_dev_run_Speaker_Diarization_with_ASR_Inference / main
L2_Speaker_dev_run_Clustering_Diarizer_Inference  /  main
1m 45s
L2_Speaker_dev_run_Clustering_Diarizer_Inference / main
L2_Speaker_dev_run_Neural_Diarizer_Inference  /  main
1m 40s
L2_Speaker_dev_run_Neural_Diarizer_Inference / main
L2_Speaker_dev_run_Multispeaker_ASR_Data_Simulation  /  main
1m 23s
L2_Speaker_dev_run_Multispeaker_ASR_Data_Simulation / main
L2_ASR_Multi-dataloader_dev_run_Speech_to_Text_multi-dataloader  /  main
53s
L2_ASR_Multi-dataloader_dev_run_Speech_to_Text_multi-dataloader / main
L2_ASR_Multi-dataloader_dev_run_Speech_to_Label_multi-dataloader  /  main
1m 29s
L2_ASR_Multi-dataloader_dev_run_Speech_to_Label_multi-dataloader / main
L2_ASR_Adapters_Linear_Adapters  /  main
40s
L2_ASR_Adapters_Linear_Adapters / main
L2_ASR_Adapters_RelPos_MHA_Adapters  /  main
1m 29s
L2_ASR_Adapters_RelPos_MHA_Adapters / main
L2_Speech_Estimate_Duration_Bins  /  main
1m 53s
L2_Speech_Estimate_Duration_Bins / main
L2_Speech_Batch_Size_OOMptimizer  /  main
2m 37s
L2_Speech_Batch_Size_OOMptimizer / main
L2_Speech_Batch_Size_OOMptimizer_Canary  /  main
2m 4s
L2_Speech_Batch_Size_OOMptimizer_Canary / main
L2_Speech_Transcription_Speech_to_Text_Transcribe  /  main
1m 27s
L2_Speech_Transcription_Speech_to_Text_Transcribe / main
L2_Speech_Transcription_Canary_Transcribe_Full_Manifest  /  main
2m 41s
L2_Speech_Transcription_Canary_Transcribe_Full_Manifest / main
L2_Speech_Transcription_Canary_Transcribe_With_Prompt  /  main
2m 42s
L2_Speech_Transcription_Canary_Transcribe_With_Prompt / main
L2_Speech_Transcription_Canary_Transcribe_Audio_Dir  /  main
2m 44s
L2_Speech_Transcription_Canary_Transcribe_Audio_Dir / main
L2_Segmentation_Tool_Parallel_ctc_segmentation_test_L2_Eng_CitriNet_with_wav  /  main
1m 13s
L2_Segmentation_Tool_Parallel_ctc_segmentation_test_L2_Eng_CitriNet_with_wav / main
L2_Segmentation_Tool_Parallel_ctc_segmentation_test_L2_Ru_QN_with_mp3  /  main
1m 13s
L2_Segmentation_Tool_Parallel_ctc_segmentation_test_L2_Ru_QN_with_mp3 / main
L2_G2P_Models_G2P_Conformer_training_evaluation_and_inference  /  main
1m 57s
L2_G2P_Models_G2P_Conformer_training_evaluation_and_inference / main
L2_G2P_Models_HeteronymClassificationModel_training_evaluation_and_inference  /  main
2m 37s
L2_G2P_Models_HeteronymClassificationModel_training_evaluation_and_inference / main
L2_Pretraining_BERT_pretraining_from_Text  /  main
1m 7s
L2_Pretraining_BERT_pretraining_from_Text / main
L2_Pretraining_BERT_from_Preprocessed  /  main
49s
L2_Pretraining_BERT_from_Preprocessed / main
L2_NMT_Attention_is_All_You_Need_Training_NMT_Training_Post-LN  /  main
1m 9s
L2_NMT_Attention_is_All_You_Need_Training_NMT_Training_Post-LN / main
L2_NMT_Attention_is_All_You_Need_Training_NMT_Training_Pre-LN  /  main
56s
L2_NMT_Attention_is_All_You_Need_Training_NMT_Training_Pre-LN / main
L2_NMT_Attention_is_All_You_Need_Training_NMT_Multi-Validation  /  main
1m 9s
L2_NMT_Attention_is_All_You_Need_Training_NMT_Multi-Validation / main
L2_NMT_Attention_is_All_You_Need_Inference  /  main
1m 54s
L2_NMT_Attention_is_All_You_Need_Inference / main
L2_NMT_Attention_is_All_You_Need_Finetuning  /  main
1m 41s
L2_NMT_Attention_is_All_You_Need_Finetuning / main
L2_NMT_Tarred_Dataset_Creation_Auto_Tarred_Dataset_Creation  /  main
1m 43s
L2_NMT_Tarred_Dataset_Creation_Auto_Tarred_Dataset_Creation / main
L2_NMT_Tarred_Dataset_Creation_Script_Tarred_Dataset_Creation  /  main
1m 42s
L2_NMT_Tarred_Dataset_Creation_Script_Tarred_Dataset_Creation / main
L2_Megatron_NMT_Training_TP2  /  main
5m 3s
L2_Megatron_NMT_Training_TP2 / main
L2_Megatron_Bert_Pretraining_and_Resume_Training_with_Pipeline_Parallelism  /  main
3m 56s
L2_Megatron_Bert_Pretraining_and_Resume_Training_with_Pipeline_Parallelism / main
L2_Megatron_Core_Bert_Pretraining_and_Resume_Training  /  main
4m 6s
L2_Megatron_Core_Bert_Pretraining_and_Resume_Training / main
L2_RAG_Pipeline_Indexing  /  main
1m 46s
L2_RAG_Pipeline_Indexing / main
L2_RAG_Pipeline_Generating  /  main
2m 24s
L2_RAG_Pipeline_Generating / main
L2_Megatron_GPT_Pretraining_and_Resume_Training_TP2  /  main
4m 19s
L2_Megatron_GPT_Pretraining_and_Resume_Training_TP2 / main
L2_Megatron_GPT_Skip_Train  /  main
2m 59s
L2_Megatron_GPT_Skip_Train / main
L2_Megatron_GPT_with_Rope_Pretraining_and_Resume_Training_TP2  /  main
3m 20s
L2_Megatron_GPT_with_Rope_Pretraining_and_Resume_Training_TP2 / main
L2_Megatron_LM_To_NeMo_Conversion  /  main
4m 11s
L2_Megatron_LM_To_NeMo_Conversion / main
L2_Megatron_GPT_with_ResetLR_Pretraining_and_Resume_Training_TP2  /  main
4m 37s
L2_Megatron_GPT_with_ResetLR_Pretraining_and_Resume_Training_TP2 / main
L2_Megatron_GPT_with_Drop_Optimizer_States_TP2  /  main
3m 35s
L2_Megatron_GPT_with_Drop_Optimizer_States_TP2 / main
L2_Megatron_GPT_with_ALiBi_Pretraining_and_Resume_Training_TP2  /  main
3m 34s
L2_Megatron_GPT_with_ALiBi_Pretraining_and_Resume_Training_TP2 / main
L2_Megatron_GPT_with_KERPLE_Pretraining_and_Resume_Training_TP2  /  main
3m 19s
L2_Megatron_GPT_with_KERPLE_Pretraining_and_Resume_Training_TP2 / main
L2_Megatron_GPT_Pretraining_and_Resume_Training_PP2  /  main
4m 10s
L2_Megatron_GPT_Pretraining_and_Resume_Training_PP2 / main
L2_Megatron_GPT_Auto_Configurator_TP1_PP1_MBS124  /  main
2m 31s
L2_Megatron_GPT_Auto_Configurator_TP1_PP1_MBS124 / main
L2_Megatron_GPT_Finetuning_PP2  /  main
4m 33s
L2_Megatron_GPT_Finetuning_PP2 / main
L2_Megatron_GPT_Finetuning_StarCoder_PP1  /  main
1m 18s
L2_Megatron_GPT_Finetuning_StarCoder_PP1 / main
L2_Megatron_GPT_Reranker  /  main
1m 43s
L2_Megatron_GPT_Reranker / main
L2_Megatron_GPT_Embedding  /  main
2m 34s
L2_Megatron_GPT_Embedding / main
L2_Megatron_GPT_PEFT_Lora_PP2_O2  /  main
2m 46s
L2_Megatron_GPT_PEFT_Lora_PP2_O2 / main
L2_Megatron_GPT_PEFT_Lora_TP2_O1  /  main
2m 48s
L2_Megatron_GPT_PEFT_Lora_TP2_O1 / main
L2_Megatron_GPT_PEFT_Lora_TP2SP1  /  main
1m 8s
L2_Megatron_GPT_PEFT_Lora_TP2SP1 / main
L2_Megatron_GPT_Eval  /  main
1m 58s
L2_Megatron_GPT_Eval / main
L2_Megatron_GPT_Eval_PP2  /  main
2m 39s
L2_Megatron_GPT_Eval_PP2 / main
L2_Megatron_GPT_SFT_Eval_inference_seq_len_greaterThan_training_seq_len  /  main
2m 2s
L2_Megatron_GPT_SFT_Eval_inference_seq_len_greaterThan_training_seq_len / main
L2_Megatron_Change_Partitions_Reduce_TP_Num_Partitions_-2_to_1-_and_PP_Num_Partitions_-1_to_2  /  main
2m 8s
L2_Megatron_Change_Partitions_Reduce_TP_Num_Partitions_-2_to_1-_and_PP_Num_Partitions_-1_to_2 / main
L2_Megatron_Change_Partitions_Increase_TP_Num_Partitions_-2_to_4-_and_PP_Num_Partitions_-1_to_2  /  main
2m 0s
L2_Megatron_Change_Partitions_Increase_TP_Num_Partitions_-2_to_4-_and_PP_Num_Partitions_-1_to_2 / main
L2_Megatron_Core_T5_Pretraining_and_Resume_Training_TP2  /  main
3m 35s
L2_Megatron_Core_T5_Pretraining_and_Resume_Training_TP2 / main
L2_Megatron_T5_with_ALiBi_Pretraining_and_Resume_Training_TP2  /  main
3m 33s
L2_Megatron_T5_with_ALiBi_Pretraining_and_Resume_Training_TP2 / main
L2_Megatron_T5_with_KERPLE_Pretraining_and_Resume_Training_TP2  /  main
3m 2s
L2_Megatron_T5_with_KERPLE_Pretraining_and_Resume_Training_TP2 / main
L2_Megatron_T5_w_Mixture_of_Expert_Pretraining  /  main
2m 11s
L2_Megatron_T5_w_Mixture_of_Expert_Pretraining / main
L2_Megatron_UL2_Pretraining_and_Resume_Training_TP2  /  main
3m 6s
L2_Megatron_UL2_Pretraining_and_Resume_Training_TP2 / main
L2_Megatron_Core_T5_Eval  /  main
1m 59s
L2_Megatron_Core_T5_Eval / main
L2_Megatron_Core_T5_PEFT_Lora_TP2  /  main
4m 2s
L2_Megatron_Core_T5_PEFT_Lora_TP2 / main
L2_VLM_HF_Transformer_PEFT  /  main
1m 24s
L2_VLM_HF_Transformer_PEFT / main
L2_VLM_HF_Transformer_PEFT_FSDP  /  main
2m 28s
L2_VLM_HF_Transformer_PEFT_FSDP / main
L2_VLM_HF_Transformer_PEFT_4bit  /  main
1m 26s
L2_VLM_HF_Transformer_PEFT_4bit / main
L2_VLM_HF_Transformer_SFT_FSDP2  /  main
43s
L2_VLM_HF_Transformer_SFT_FSDP2 / main
L2_HF_Transformer_PEFT  /  main
1m 25s
L2_HF_Transformer_PEFT / main
L2_HF_Transformer_PEFT_nemorun  /  main
1m 56s
L2_HF_Transformer_PEFT_nemorun / main
L2_HF_Transformer_PEFT_2gpu  /  main
2m 36s
L2_HF_Transformer_PEFT_2gpu / main
L2_HF_Transformer_PEFT_2gpu_nemorun  /  main
3m 0s
L2_HF_Transformer_PEFT_2gpu_nemorun / main
L2_HF_Transformer_SFT_2gpu  /  main
6m 42s
L2_HF_Transformer_SFT_2gpu / main
L2_HF_Transformer_SFT_FSDP2_2gpu  /  main
1m 48s
L2_HF_Transformer_SFT_FSDP2_2gpu / main
L2_HF_Transformer_PT_2gpu  /  main
5m 54s
L2_HF_Transformer_PT_2gpu / main
L2_HF_Transformer_SFT_2gpu_nemorun  /  main
3m 15s
L2_HF_Transformer_SFT_2gpu_nemorun / main
L2_HF_Transformer_SFT_2gpu_nemorun_fsdp2  /  main
1m 39s
L2_HF_Transformer_SFT_2gpu_nemorun_fsdp2 / main
L2_HF_Transformer_PT_2gpu_nemorun  /  main
2m 39s
L2_HF_Transformer_PT_2gpu_nemorun / main
L2_HF_Transformer_PT  /  main
4m 51s
L2_HF_Transformer_PT / main
L2_HF_Transformer_PT_nemorun  /  main
2m 24s
L2_HF_Transformer_PT_nemorun / main
L2_HF_Transformer_SFT  /  main
5m 1s
L2_HF_Transformer_SFT / main
L2_HF_Transformer_SFT_nemorun  /  main
3m 1s
L2_HF_Transformer_SFT_nemorun / main
L2_HF_Transformer_SFT_TE_Acceleration  /  main
7m 3s
L2_HF_Transformer_SFT_TE_Acceleration / main
L2_HF_Transformer_PT_TE_Acceleration  /  main
5m 20s
L2_HF_Transformer_PT_TE_Acceleration / main
L2_HF_Transformer_SpeechLM_SFT_2gpu  /  main
2m 27s
L2_HF_Transformer_SpeechLM_SFT_2gpu / main
L2_Megatron_Mock_Data_Generation_MockGPTDataset  /  main
3m 9s
L2_Megatron_Mock_Data_Generation_MockGPTDataset / main
L2_Megatron_Mock_Data_Generation_MockT5Dataset  /  main
2m 5s
L2_Megatron_Mock_Data_Generation_MockT5Dataset / main
L2_TTS_Fast_dev_runs_1_Tacotron_2  /  main
55s
L2_TTS_Fast_dev_runs_1_Tacotron_2 / main
L2_TTS_Fast_dev_runs_1_WaveGlow  /  main
1m 41s
L2_TTS_Fast_dev_runs_1_WaveGlow / main
L2_TTS_Fast_dev_runs_1_FastPitch  /  main
1m 48s
L2_TTS_Fast_dev_runs_1_FastPitch / main
L2_TTS_Fast_dev_runs_1_Hifigan  /  main
1m 37s
L2_TTS_Fast_dev_runs_1_Hifigan / main
Speech_Checkpoints_tests  /  main
2m 17s
Speech_Checkpoints_tests / main
L2_Stable_Diffusion_Training  /  main
4m 6s
L2_Stable_Diffusion_Training / main
L2_NeMo_2_GPT_Pretraining_no_transformer_engine  /  main
4m 6s
L2_NeMo_2_GPT_Pretraining_no_transformer_engine / main
L2_NeMo_2_llama3_pretraining_recipe  /  main
4m 27s
L2_NeMo_2_llama3_pretraining_recipe / main
L2_NeMo_2_llama3_fault_tolerance_plugin  /  main
7m 32s
L2_NeMo_2_llama3_fault_tolerance_plugin / main
L2_NeMo_2_llama3_straggler_detection  /  main
2m 45s
L2_NeMo_2_llama3_straggler_detection / main
L2_NeMo_2_GPT_DDP_Param_Parity_check  /  main
2m 56s
L2_NeMo_2_GPT_DDP_Param_Parity_check / main
L2_NeMo_2_SSM_Pretraining  /  main
3m 53s
L2_NeMo_2_SSM_Pretraining / main
L2_NeMo_2_SSM_Finetuning  /  main
4m 16s
L2_NeMo_2_SSM_Finetuning / main
L2_NeMo_2_HF_MODEL_IMPORT  /  main
4m 34s
L2_NeMo_2_HF_MODEL_IMPORT / main
L2_NeMo_2_jit_callback  /  main
1m 39s
L2_NeMo_2_jit_callback / main
L2_NeMo_2_T5_Pretraining  /  main
5m 46s
L2_NeMo_2_T5_Pretraining / main
L2_NeMo_2_T5_Finetuning  /  main
4m 19s
L2_NeMo_2_T5_Finetuning / main
L2_NeMo_2_T5_LoRA  /  main
4m 24s
L2_NeMo_2_T5_LoRA / main
L2_NeMo_2_NEVA_MOCK_TRAINING  /  main
2m 30s
L2_NeMo_2_NEVA_MOCK_TRAINING / main
L2_NeMo_2_NEVA_MOCK_PACKED_TRAINING  /  main
2m 38s
L2_NeMo_2_NEVA_MOCK_PACKED_TRAINING / main
L2_NeMo_2_MLLAMA_MOCK_TRAINING  /  main
7m 44s
L2_NeMo_2_MLLAMA_MOCK_TRAINING / main
L2_NeMo_2_Mixtral_Pretraining  /  main
1m 55s
L2_NeMo_2_Mixtral_Pretraining / main
L2_NeMo_2_GPT_SFT_TP1PP1_MBS1  /  main
4m 20s
L2_NeMo_2_GPT_SFT_TP1PP1_MBS1 / main
L2_NeMo_2_GPT_SFT_TP1PP1_MBS2  /  main
4m 14s
L2_NeMo_2_GPT_SFT_TP1PP1_MBS2 / main
L2_NeMo_2_GPT_SFT_TP1PP2_MBS2  /  main
4m 29s
L2_NeMo_2_GPT_SFT_TP1PP2_MBS2 / main
L2_NeMo_2_GPT_SFT_TP2PP1_MBS2  /  main
4m 19s
L2_NeMo_2_GPT_SFT_TP2PP1_MBS2 / main
L2_NeMo_2_GPT_SFT_TP1PP1_MBS1_PACKED  /  main
4m 1s
L2_NeMo_2_GPT_SFT_TP1PP1_MBS1_PACKED / main
L2_NeMo_2_GPT_LoRA_TP1PP1_MBS1  /  main
3m 40s
L2_NeMo_2_GPT_LoRA_TP1PP1_MBS1 / main
L2_NeMo_2_GPT_LoRA_TP1PP1_MBS2  /  main
3m 55s
L2_NeMo_2_GPT_LoRA_TP1PP1_MBS2 / main
L2_NeMo_2_GPT_LoRA_TP1PP2_MBS2  /  main
3m 52s
L2_NeMo_2_GPT_LoRA_TP1PP2_MBS2 / main
L2_NeMo_2_GPT_LoRA_TP2PP1_MBS2  /  main
3m 47s
L2_NeMo_2_GPT_LoRA_TP2PP1_MBS2 / main
L2_NeMo_2_GPT_LoRA_TP1PP1_MBS1_PACKED  /  main
3m 18s
L2_NeMo_2_GPT_LoRA_TP1PP1_MBS1_PACKED / main
L2_NeMo_2_GPT_DoRA_TP1PP1_MBS1_PACKED  /  main
3m 22s
L2_NeMo_2_GPT_DoRA_TP1PP1_MBS1_PACKED / main
L2_NeMo_2_GPT_CLoRA_TP1PP1_MBS1_PACKED  /  main
3m 24s
L2_NeMo_2_GPT_CLoRA_TP1PP1_MBS1_PACKED / main
L2_NeMo_2_GPT_LoRA_TP1PP1_MBS1_Chat  /  main
3m 51s
L2_NeMo_2_GPT_LoRA_TP1PP1_MBS1_Chat / main
L2_NeMo_2_Mixtral_LoRA_EP2PP1_MBS2  /  main
2m 17s
L2_NeMo_2_Mixtral_LoRA_EP2PP1_MBS2 / main
L2_NeMo_2_Mixtral_LoRA_TP1PP1_MBS1  /  main
2m 13s
L2_NeMo_2_Mixtral_LoRA_TP1PP1_MBS1 / main
L2_NeMo_2_Mixtral_LoRA_TP2PP1_MBS1  /  main
2m 29s
L2_NeMo_2_Mixtral_LoRA_TP2PP1_MBS1 / main
L2_NeMo_2_Mistral_LoRA_TP1PP1_MBS1  /  main
2m 9s
L2_NeMo_2_Mistral_LoRA_TP1PP1_MBS1 / main
L2_NeMo_2_Mistral_LoRA_TP2PP1_MBS1  /  main
2m 31s
L2_NeMo_2_Mistral_LoRA_TP2PP1_MBS1 / main
L2_NEMO_2_LoRA_MERGE  /  main
1m 45s
L2_NEMO_2_LoRA_MERGE / main
L2_NEMO_2_LoRA_Export  /  main
38s
L2_NEMO_2_LoRA_Export / main
L2_NEMO_2_LoRA_Inference  /  main
47s
L2_NEMO_2_LoRA_Inference / main
L2_NeMo_2_NeMo_Mcore_Mixtral_bitexact  /  main
2m 52s
L2_NeMo_2_NeMo_Mcore_Mixtral_bitexact / main
L2_NeMo_2_PTQ_Llama2_FP8  /  main
2m 44s
L2_NeMo_2_PTQ_Llama2_FP8 / main
L2_NeMo_2_Distill_Llama3_TP1PP2  /  main
2m 28s
L2_NeMo_2_Distill_Llama3_TP1PP2 / main
L2_NeMo_2_Export_In_Framework  /  main
2m 56s
L2_NeMo_2_Export_In_Framework / main
L2_NeMo_2_LLAVA_NEXT_MOCK_TRAINING  /  main
1m 41s
L2_NeMo_2_LLAVA_NEXT_MOCK_TRAINING / main
L2_NeMo_2_VLLM_EXPORT  /  main
3m 48s
L2_NeMo_2_VLLM_EXPORT / main
L0_Setup_Test_Data_And_Models  /  main
1m 28s
L0_Setup_Test_Data_And_Models / main
OPTIONAL_L2_Megatron_T5_Pretraining_and_Resume_Training_PP2  /  main
3m 16s
OPTIONAL_L2_Megatron_T5_Pretraining_and_Resume_Training_PP2 / main
L0_Unit_Tests_Coverage
1m 16s
L0_Unit_Tests_Coverage
Nemo_CICD_Test
2s
Nemo_CICD_Test
Fit to window
Zoom out
Zoom in

Annotations

1 error
OPTIONAL_L2_Megatron_T5_Pretraining_and_Resume_Training_PP2 / main
Process completed with exit code 1.

Artifacts

Produced during runtime
Name Size
coverage-13208247505-033bfbf4-63ea-4db2-85ae-2ef8bad112d7
1.1 MB
coverage-13208247505-0e0d2708-dc0f-4dd6-aed5-8a8e520dde45
695 KB
coverage-13208247505-161bc171-4aee-4f66-9543-632b1c7778ce
1.52 MB
coverage-13208247505-34c952ab-7747-4175-855b-5afed15a414e
1.59 MB
coverage-13208247505-3864a746-721f-48a6-aefd-774cd1a5a43b
1.11 MB
coverage-13208247505-465121ea-0db1-4975-aadd-35693ed0bb97
1.59 MB
coverage-13208247505-5c440066-4712-4f0f-b0e8-5d81bd2b79bb
1.04 MB
coverage-13208247505-79c99b56-62d4-48be-beaa-5c4f920ee2cf
1.47 MB
coverage-13208247505-81dcae1f-ed6d-4656-a92b-34102d0ceded
1.09 MB
coverage-13208247505-844645b7-bc80-4b01-8593-e28216506ce7
1.59 MB
coverage-13208247505-905d3a94-3b76-4780-9468-e97c2cb0dfcb
1.49 MB
coverage-13208247505-963e7c09-7ff9-476f-a70a-36fb7aaaff3d
1.37 MB
coverage-13208247505-9ade3e46-1e12-4b04-ac67-057d74db5886
1.49 MB
coverage-13208247505-aa23a213-e5d7-4f75-92dd-ef2c99edb6ca
1.52 MB
coverage-13208247505-b5563c14-d35d-457a-8372-c6c352d0541d
1.47 MB
coverage-13208247505-b64ae3b1-03f2-4c78-8ccb-3bca8097d023
1.08 MB
coverage-13208247505-be52514b-195a-40e8-aac7-70b8f8af67f0
1.03 MB
coverage-13208247505-caeea610-c71e-4a54-930f-956ba9d75b74
1.48 MB
coverage-13208247505-dd3cc8cb-8a84-4a2d-b5ad-c6032b1f4399
693 KB
coverage-13208247505-e43ee999-f2d4-4e44-89c2-dcbe7511f7d4
1.53 MB
coverage-13208247505-fcbcd7a1-f846-4eed-a12e-842c4fc8acf7
1.48 MB
coverage-summary-13208247505
1.86 MB