[automodel] fix loss/tps reporting across ranks #9314

Run time

Learn about OS pricing on GitHub Actions

Job	Run time
pre-flight	9s
cicd-test-container-build / main	7m 20s
cicd-import-tests	1m 53s
L0_Unit_Tests_GPU_ASR / main	13m 49s
L0_Unit_Tests_GPU_Audio / main	2m 57s
L0_Unit_Tests_GPU_Common / main	6m 10s
L0_Unit_Tests_GPU_LLM / main	4m 58s
L0_Unit_Tests_GPU_Core / main	6m 49s
L0_Unit_Tests_GPU_TTS / main	2m 18s
L0_Unit_Tests_GPU_Hydra / main	6m 5s
L0_Unit_Tests_CPU_ASR / main	9m 59s
L0_Unit_Tests_GPU_Multimodal / main	3m 1s
L0_Unit_Tests_GPU_Lightning / main	4m 45s
L0_Unit_Tests_GPU_Others / main	6m 13s
L0_Unit_Tests_CPU_Common / main	3m 28s
L0_Unit_Tests_CPU_Audio / main	2m 11s
L0_Unit_Tests_CPU_Core / main	7m 57s
L0_Unit_Tests_CPU_Hydra / main	3m 42s
L0_Unit_Tests_CPU_LLM / main	2m 32s
L0_Unit_Tests_CPU_Multimodal / main	2m 21s
L0_Unit_Tests_CPU_TTS / main	3m 3s
ASR_dev_run_Speech_to_Text / main	2m 34s
L0_Setup_Test_Data_And_Models / main	48s
ASR_dev_run_Speech_Pre-training_-_CitriNet / main	1m 24s
L0_Unit_Tests_CPU_Lightning / main	3m 3s
L0_Unit_Tests_CPU_Others / main	2m 57s
L2_Community_vita_Checkpoints_tests_Llama3 / main	5m 47s
ASR_dev_run_Speech_To_Text_Finetuning / main	2m 30s
ASR_dev_run_Speech_to_Text_WPE_-_CitriNet / main	2m 35s
ASR_dev_run_Speech_To_Text_HF_Finetuning / main	2m 22s
ASR_dev_run_Speech_to_Text_WPE_-_Conformer / main	2m 22s
L2_Speaker_dev_run_EndtoEnd_Speaker_Diarization_Sortformer / main	1m 27s
ASR_dev_run-part_two_Speech_to_Text_WPE_-_Squeezeformer / main	2m 47s
L2_Speaker_dev_run_Speaker_Recognition / main	2m 15s
L2_Speech_to_Text_EMA / main	1m 34s
L2_Speaker_dev_run_Clustering_Diarizer_Inference / main	1m 30s
L2_Speaker_dev_run_Speaker_Diarization / main	2m 20s
L2_Speech_to_Text_AED / main	1m 32s
L2_ASR_Multi-dataloader_dev_run_Speech_to_Text_multi-dataloader / main	2m 36s
L2_Speaker_dev_run_EndtoEnd_Diarizer_Inference / main	1m 12s
L2_Speaker_dev_run_Multispeaker_ASR_Data_Simulation / main	1m 12s
L2_Speaker_dev_run_Neural_Diarizer_Inference / main	1m 33s
L2_Speaker_dev_run_Speech_to_Label / main	2m 17s
L2_Speaker_dev_run_Speaker_Diarization_with_ASR_Inference / main	1m 16s
L2_ASR_Adapters_Linear_Adapters / main	2m 24s
L2_ASR_Adapters_RelPos_MHA_Adapters / main	2m 21s
L2_ASR_Multi-dataloader_dev_run_Speech_to_Label_multi-dataloader / main	1m 26s
L2_Speech_Estimate_Duration_Bins / main	1m 31s
L2_Speech_Batch_Size_OOMptimizer / main	1m 57s
L2_Speech_Transcription_Canary_Transcribe_Full_Manifest / main	2m 12s
L2_G2P_Models_G2P_Conformer_training_evaluation_and_inference / main	1m 42s
L2_Segmentation_Tool_Parallel_ctc_segmentation_test_L2_Eng_CitriNet_with_wav / main	2m 17s
L2_Speech_Transcription_Speech_to_Text_Transcribe / main	1m 19s
Optional_L2_Speech_Batch_Size_OOMptimizer_Canary / main	1m 41s
L2_Segmentation_Tool_Parallel_ctc_segmentation_test_L2_Ru_QN_with_mp3 / main	1m 54s
L2_G2P_Models_HeteronymClassificationModel_training_evaluation_and_inference / main	3m 25s
L2_Speech_Transcription_Canary_Transcribe_Audio_Dir / main	3m 22s
L2_NMT_Attention_is_All_You_Need_Finetuning / main	4m 16s
L2_NMT_Attention_is_All_You_Need_Training_NMT_Training_Post-LN / main	4m 58s
L2_NMT_Attention_is_All_You_Need_Inference / main	3m 18s
L2_Speech_Transcription_Canary_Transcribe_With_Prompt / main	2m 13s
L2_Megatron_NMT_Training_TP2 / main	4m 49s
L2_NMT_Attention_is_All_You_Need_Training_NMT_Multi-Validation / main	2m 30s
L2_NMT_Attention_is_All_You_Need_Training_NMT_Training_Pre-LN / main	1m 56s
L2_NMT_Tarred_Dataset_Creation_Auto_Tarred_Dataset_Creation / main	1m 44s
L2_HF_Transformer_PEFT / main	2m 16s
L2_NMT_Tarred_Dataset_Creation_Script_Tarred_Dataset_Creation / main	1m 59s
L2_VLM_HF_Transformer_PEFT_FSDP2 / main	28s
L2_HF_Transformer_PEFT_2gpu / main	1m 47s
L2_HF_Transformer_PEFT_2gpu_FSDP2 / main	1m 16s
L2_HF_Transformer_PEFT_2gpu_nemorun / main	1s
L2_HF_Transformer_PEFT_notebook / main	1s
L2_HF_Transformer_PEFT_nemorun / main	1s
L2_VLM_HF_Transformer_PEFT / main	1s
L2_VLM_HF_Transformer_PEFT_4bit / main	1s
L2_VLM_HF_Transformer_SFT_FSDP2 / main	1s
L2_HF_Transformer_PT / main	1s
L2_HF_Transformer_PT_2gpu / main	1s
L2_HF_Transformer_PT_2gpu_nemorun / main	1s
L2_HF_Transformer_PT_TE_Acceleration / main	1s
L2_HF_Transformer_PT_nemorun / main	1s
L2_HF_Transformer_SFT / main	1s
L2_HF_Transformer_SFT_2gpu / main	1s
L2_HF_Transformer_SFT_2gpu_FSDP2 / main	1s
L2_HF_Transformer_SFT_2gpu_nemorun / main	1s
L2_HF_Transformer_SFT_2gpu_nemorun_fsdp2 / main	1s
L2_HF_Transformer_SFT_FSDP2_2gpu / main	1s
L2_HF_Transformer_SFT_nemorun / main	1s
L2_HF_Transformer_SFT_TE_Acceleration / main	1s
L2_HF_Transformer_SpeechLM_SFT_2gpu / main	1s
L2_HF_Transformer_SFT_notebook / main	1s
L2_Stable_Diffusion_Training / main	1s
L2_TTS_Fast_dev_runs_1_Tacotron_2 / main	1s
L2_TTS_Fast_dev_runs_1_Hifigan / main	1s
L2_NeMo_2_GPT_DDP_Param_Parity_check / main	1s
L2_NeMo_2_HF_MODEL_IMPORT / main	1s
Speech_Checkpoints_tests / main	1s
L2_TTS_Fast_dev_runs_1_FastPitch / main	1s
L2_TTS_Fast_dev_runs_1_WaveGlow / main	1s
L2_NeMo_2_GPT_Pretraining_no_transformer_engine / main	1s
L2_NeMo_2_llama3_fault_tolerance_plugin / main	1s
L2_NeMo_2_llama3_pretraining_recipe / main	1s
L2_NeMo_2_llama3_straggler_detection / main	1s
L2_NeMo_2_jit_callback / main	1s
L2_NeMo_2_NEVA_MOCK_TRAINING / main	1s
L2_NeMo_2_SSM_Finetuning / main	1s
L2_NeMo_2_GPT_SFT_TP1PP1_MBS1 / main	1s
L2_NeMo_2_SSM_Pretraining / main	1s
L2_NeMo_2_T5_Finetuning / main	1s
L2_NeMo_2_T5_LoRA / main	1s
L2_NeMo_2_GPT_SFT_TP1PP1_MBS1_PACKED / main	1s
L2_NeMo_2_T5_Pretraining / main	1s
L2_NeMo_2_GPT_SFT_TP1PP2_MBS2 / main	1s
L2_NeMo_2_GPT_SFT_TP1PP1_MBS2 / main	1s
L2_NeMo_2_Mixtral_Pretraining / main	1s
L2_NeMo_2_GPT_SFT_TP2PP1_MBS2 / main	1s
L2_NeMo_2_MLLAMA_MOCK_TRAINING / main	1s
L2_NeMo_2_NEVA_MOCK_PACKED_TRAINING / main	1s
L2_NeMo_2_GPT_LoRA_TP1PP1_MBS1 / main	1s
L2_NeMo_2_GPT_DoRA_TP1PP1_MBS1_PACKED / main	1s
L2_NeMo_2_GPT_LoRA_TP1PP1_MBS1_PACKED / main	1s
L2_NeMo_2_GPT_LoRA_TP1PP1_MBS2 / main	1s
L2_NeMo_2_GPT_LoRA_TP1PP2_MBS2 / main	1s
L2_NeMo_2_GPT_LoRA_TP2PP1_MBS2 / main	1s
L2_NeMo_2_GPT_LoRA_TP1PP1_MBS1_Chat / main	1s
L2_NeMo_2_GPT_CLoRA_TP1PP1_MBS1_PACKED / main	1s
L2_NeMo_2_Mixtral_LoRA_EP2PP1_MBS2 / main	1s
L2_NEMO_2_LoRA_Export / main	1s
L2_NeMo_2_Mixtral_LoRA_EP2PP1_MBS2_exclude / main	1s
L2_NeMo_2_Mixtral_LoRA_TP1PP1_MBS1 / main	1s
L2_NEMO_2_LoRA_Inference / main	1s
L2_NeMo_2_Mistral_LoRA_TP1PP1_MBS1 / main	1s
L2_NEMO_2_LoRA_MERGE / main	1s
L2_NeMo_2_Mistral_LoRA_TP1PP1_MBS1_exclude / main	1s
L2_NeMo_2_Mistral_LoRA_TP2PP1_MBS1 / main	1s
L2_NeMo_2_Mixtral_LoRA_TP2PP1_MBS1 / main	1s
L2_NeMo_2_NeMo_Mcore_Mixtral_bitexact / main	1s
L2_NeMo_2_Export_In_Framework / main	1s
L2_NeMo_2_Distill_Llama3_TP1PP2 / main	1s
L2_NeMo_2_PTQ_Llama2_FP8_nemo / main	1s
L2_NeMo_2_LLAVA_NEXT_MOCK_TRAINING / main	1s
L2_NeMo_2_Auto_Configurator_TP1_PP1_MBS124 / main	1s
L2_NeMo_2_EVAL / main	1s
L2_NeMo_2_PTQ_Llama2_FP8_trtllm / main	1s
L2_NeMo_2_VLLM_EXPORT / main	1s
L2_SpeechLM_LoRA_TP1PP1_MBS2 / main	1s
Nemo_CICD_Test	13s
Coverage	1s
	3h 34m 4s

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[automodel] fix loss/tps reporting across ranks #9314

Usage

[automodel] fix loss/tps reporting across ranks #9314

Jobs

Run details

Run time