v0.9: Mixtral, Phi, Zephyr, and text classification for LLMs

What's Changed

• int: Rename original combiner_registry to combiner_config_registry, update decorator name by @ksbrar in #3516
• Add mechanic to override default values for generation during model.predict() by @justinxzhao in #3520
• [feat] Support for numeric date feature inputs by @jeffkinnison in #3517
• Add new sythesized response column for text output features during postprocessing by @arnavgarg1 in #3521
• Disable flaky twitter bots dataset loading test. by @justinxzhao in #3439
• Add test that verifies that the generation config passed in at model.predict() is used correctly. by @justinxzhao in #3523
• Move loss metric to same device as inputs by @Infernaught in #3522
• Add comment about batch size tuning by @arnavgarg1 in #3526
• Ensure user sets backend to local w/ quantization by @Infernaught in #3524
• README: Update LLM fine-tuning config by @arnavgarg1 in #3530
• Revert "Ensure user sets backend to local w/ quantization (#3524)" by @tgaddair in #3531
• Improve observability during LLM inference by @arnavgarg1 in #3536
• [bug] Pin pydantic to < 2.0 by @jeffkinnison in #3537
• [bug] Support preprocessing datetime.date date features by @jeffkinnison in #3534
• Remove obsolete prompt tuning example. by @justinxzhao in #3540
• Add Ludwig 0.8 notebook to the README by @arnavgarg1 in #3542
• Add effective_batch_size to auto-adjust gradient accumulation by @tgaddair in #3533
• Refactor evaluation metrics to support decoded generated text metrics like BLEU and ROUGE. by @justinxzhao in #3539
• Fix sequence generator test. by @justinxzhao in #3546
• Revert "Add Cosine Annealing LR scheduler as a decay method (#3507)" by @justinxzhao in #3545
• Set default max_sequence_length to None for LLM text input/output features by @arnavgarg1 in #3547
• Add skip_all_evaluation as a mechanic to skip all evaluation. by @justinxzhao in #3543
• Roll-forward with fixes: Fix interaction between scheduler.step() and gradient accumulation steps, refactor schedulers to use LambdaLR, and add cosine annealing LR scheduler as a decay method. by @justinxzhao in #3555
• fix: Move model to the correct device for eval by @jeffkinnison in #3554
• Report loss in tqdm to avoid log spam by @tgaddair in #3559
• Wrap each metric update in try/except. by @justinxzhao in #3562
• Move DDP model to device if it hasn't been wrapped yet by @tgaddair in #3566
• ensure that there are enough colors to match the score index in visua… by @thelinuxkid in #3560
• Pin Transformers to 4.31.0 by @arnavgarg1 in #3569
• Add test to show global_max_sequence_length can never exceed an LLMs context length by @arnavgarg1 in #3548
• WandB: Add metric logging support on eval end and epoch end by @arnavgarg1 in #3586
• schema: Add prompt validation check by @ksbrar in #3564
• Unpin Transformers for CodeLlama support by @arnavgarg1 in #3592
• Add support for Paged Optimizers (Adam, Adamw), 8-bit optimizers, and new optimizers: LARS, LAMB and LION by @arnavgarg1 in #3588
• FIX: Failure in TabTransformer Combiner Unit test by @jimthompson5802 in #3596
• fix: Move target tensor to model output device in check_module_parameters_updated by @jeffkinnison in #3567
• Allow user to specify huggingface link or local path to pretrained lora weights by @Infernaught in #3572
• Add codellama to tokenizer list for set_pad_token by @Infernaught in #3598
• Set default eval batch size to 2 for LLM fine-tuning by @arnavgarg1 in #3599
• [CI] Explicitly set eval batch size in determinism tests, introduce a new integration test group, and exclude slow tests. by @justinxzhao in #3590
• [CI] Run sudo apt-get update in GHAs. by @justinxzhao in #3608
• Store steps_per_epoch in Trainer by @hungcs in #3601
• Updated characters, underscore and comma preprocessors to be TorchScriptable. by @martindavis in #3602
• [CI] Deflake: Explicitly set eval batch size for mlflow test. by @justinxzhao in #3612
• Fix registration for char error rate. by @justinxzhao in #3604
• fix: Load 8-bit quantized models for eval after fine-tuning by @jeffkinnison in #3606
• Add Code Alpaca and Consumer Complaints Datasets by @connor-mccorm in #3611
• Add support for gradient checkpointing for LLM fine-tuning by @arnavgarg1 in #3613
• Bump min support transformers to 4.33.0 by @tgaddair in #3616
• [CI] Fix failing tests on master by @arnavgarg1 in #3617
• Eliminate short-circuiting for loading from local by @Infernaught in #3600
• Refactor integration tests into matrix by @tgaddair in #3618
• fix: Check underlying model device type when moving 8-bit quantized models to GPU at eval by @jeffkinnison in #3622
• Fixed range validation for text generation penalty parameters by @tgaddair in #3623
• Update comment for predict to update Ludwig docs by @Infernaught in #3535
• Avoid deprecation warnings on pandas Series.fillna by @carlogrisetti in #3631
• QoL: Default to using fast tokenizer for Llama models by @arnavgarg1 in #3625
• fixed typo in EfficientNet's model variant from v2_ to v2_s by @saad-palapa in #3628
• Add pytorch profiler and additional tensorboard logs for GPU memory usage. by @justinxzhao in #3607
• Pin minimum transformers version to 4.33.2 by @arnavgarg1 in #3637
• Add function to free GPU memory by @Infernaught in #3643
• ❗ Enable LLM fine-tuning tests when no quantization is specified by @arnavgarg1 in #3626
• Add check to ensure selected backend works with quantization for LLMs by @arnavgarg1 in #3646
• [CI] Use a torch-nightly-compatible version of torchaudio by @justinxzhao in #3644
• Set do_sample default to True by @Infernaught in #3641
• FIX: Failure in audio feature related test by @jimthompson5802 in #3651
• Remove unnecessary peft config updating by @Infernaught in #3642
• FIX: docker build error for ludwig-gpu by @jimthompson5802 in #3658
• Exclude getdaft on Windows by @carlogrisetti in #3629
• Add daft back for windows since the wheels are now officially published by @arnavgarg1 in #3663
• fix: The final batch of an epoch is skipped when batch size is 1 by @jeffkinnison in #3653
• Place metric functions for BLEU and Rogue on correct devices when using multiple GPUs by @arnavgarg1 in #3671
• Remove duplicate metrics by @Infernaught in #3670
• Increment epochs based on last_batch() instead of at the end of the train loop. by @justinxzhao in #3668
• [FEATURE] Support Merging LoRA Weights Into Base Model (Issue-3603) by @alexsherstinsky in #3649
• [FEATURE] Include Mistral-7B model in list of supported base models by @alexsherstinsky in #3674
• [MAINTENANCE] Partially reconcile type hints, fix some warnings, and fix comments in parts of the codebase. by @alexsherstinsky in #3673
• Improve error message for when an LLM base model can't be loaded. by @justinxzhao in #3675
• Fix eos_token and pad_token issue by @Infernaught in #3667
• FIX: error with nightly CI tests for test_resize_image by @jimthompson5802 in #3678
• [BUGFIX] Remove spurious test directory at the end of the test_llm.py::test_local_path_loading test run by @alexsherstinsky in #3680
• Add per-device logging to tensorboard by @Infernaught in #3677
• Fix dynamic generation config load during model.predict by @geoffreyangus in #3666
• [CI] Ensure that mlflow callback cleans up background-saving threads on trainer teardown. by @justinxzhao in #3683
• fix: temporarily remove config validation check for backend by @geoffreyangus in #3688
• fix: Failing test for backend with quantization by @arnavgarg1 in #3689
• [BUGFIX] Ensure that full base models and not only adapter weights get saved when merge_and_unload is set by @alexsherstinsky in #3679
• Add Ludwig Star History to README by @arnavgarg1 in #3696
• Use sphinx for all docstrings in api.py by @justinxzhao in #3693
• Fix binary variables being visualized as 0 and 1 by @Infernaught in #3691
• [MAINTENANCE] Fix the linting warnings in two backend component classes. by @alexsherstinsky in #3698
• [BUGFIX] Pin deepspeed<0.11, skip Horovod tests by @alexsherstinsky in #3700
• Unpin deepspeed following fix in v0.11.1 by @tgaddair in #3706
• Move on_epoch_end and epoch increment to after run_evaluation loop. by @justinxzhao in #3690
• Remove model_load_path from experiment by @Infernaught in #3707
• [FEATURE] Allow typehints without the quotes. by @alexsherstinsky in #3699
• Add consumer complaints generation dataset by @connor-mccorm in #3685
• Set the metadata only during first training run by @Infernaught in #3684
• Add ability to upload Ludwig models to Predibase. by @martindavis in #3687
• Log additional per-GPU information in model metadata files and GPU utilization on tensorboard. by @justinxzhao in #3712
• QoL: Only log generation config being used once at inference time by @arnavgarg1 in #3715
• [MAINTENANCE] Adding typehint annotations in backend and data components and fixing mypy errors. by @alexsherstinsky in #3709
• QoL: Limit top-level trainer logging messages such as saving model or resuming model training to main coordinator process by @arnavgarg1 in #3718
• Add sample_size as a global preprocessing parameter by @Infernaught in #3650
• QOL: Update recommended vscode settings. by @justinxzhao in #3717
• Add new fine-tuning notebooks to README by @arnavgarg1 in #3722
• Dynamically set max_new_tokens based on output feature length, GMSL and model window size by @arnavgarg1 in #3713
• Fix issue while logging cuda device utilization to tensorboard by @arnavgarg1 in #3727
• Pin PEFT to PEFT master for Mistral-7B support by @arnavgarg1 in #3725
• Limit ray versions to keep 2.4 out of the picture. by @trevenrawr in #3731
• Set default text preprocessing lowercase behavior to False by @connor-mccorm in #3721
• Add functionality to allow direct importing of datasets from Hugging Face by @Infernaught in #3664
• [MAINTENACE] Improve typehints and use constants instead of common hardcoded string values. by @alexsherstinsky in #3732
• DRAFT: Call .finish() for WandB callback on trainer end. by @justinxzhao in #3710
• Fix calculation of dataset size by @Infernaught in #3723
• [BUGFIX] Make sure that the "hyperopt" directory in the local repository path is deleted after test run is completed by @alexsherstinsky in #3736
• [BUGFIX] Making integration tests in the group "integration_tests_e" pass in Azure CI by @alexsherstinsky in #3737
• [CI] Fix test broken by transformers v.4.34.1 by @justinxzhao in #3739
• Raise error if output feature name is in the prompt template by @arnavgarg1 in #3743
• Retry LLM model downloads from HF Hub with exponential backoff when there is a Read timeout by @arnavgarg1 in #3742
• Simplify how we set pad token and pad token ID for huggingface tokenizers. by @justinxzhao in #3735
• Increment the epoch before running evaluation by @jeffkinnison in #3729
• Uncap the torchmetrics version, so we can get fixes in 1.0+. by @trevenrawr in #3741
• Fix Consumer Complaints Gen Dataset Config by @connor-mccorm in #3746
• [MAINTENANCE] Restrict "fsspec" to not include the latest version "2023.10.0" by @alexsherstinsky in #3752
• [MAINTENACE] Fixing method overload violations; providing typehints and method argument documentation by @alexsherstinsky in #3753
• Fix: Prevent memory from ballooning during post-training evaluation by @arnavgarg1 in #3756
• Update base.py by @jetaudio in #3750
• Pin getdaft<0.2 in requirements by @jaychia in #3757
• QOL: Fail during preprocessing if max sequence lengths are shorter than the prompt template. by @justinxzhao in #3719
• Define initialize_llm() and generate() methods. Remove extra logging in llm.py by @justinxzhao in #3711
• Update ludwig version to 0.9.dev. by @justinxzhao in #3638
• fix: Check device type in _initialize_llm before moving to cuda. by @justinxzhao in #3762
• [BUGFIX] Pin Transformers version to prevent tests from failing in PEFT by @alexsherstinsky in #3766
• [MAINTENANCE] Update contributor instructions with additional steps that may be needed for installation to succeed. by @alexsherstinsky in #3764
• fix: adapter checkpoint loading on resume by @geoffreyangus in #3769
• fix LudwigProgressBar rendering on model resume by @geoffreyangus in #3770
• enh: Add on_checkpoint callback that triggers after every checkpoint even if the model was not evaluated by @jeffreyftang in #3763
• [BUGFIX] Un-pin "transformers" version and update file names for saved merged model directory, because issue with "transformers" impacting "peft" has been fixed. by @alexsherstinsky in #3775
• OnnxExporter by @skanjila in #3761
• [CI] Upgrade torch on CIs to 2.0.0, 2.1.1, and nightly by @justinxzhao in #3785
• Cleanup: Use existing HFTokenizer to consolidate manual pad token setting. by @justinxzhao in #3774
• Update copyright by @geoffreyangus in #3778
• Add k eval predictions to mlflow for LLM fine-tuning by @Infernaught in #3768
• Add eval_steps as a flag by @Infernaught in #3776
• Fix MLFlow logging for sample outputs during evaluation. by @arnavgarg1 in #3786
• Add NEFTune implementation for Noised Embedding Instruction Fine-Tuning support by @arnavgarg1 in #3744
• [MAINTENANCE] Provide links to contributor guide in README.md by @alexsherstinsky in #3794
• [Troubleshooting] [BUGFIX] Troubleshooting the "slow" tests group in Azure DevOps CI. by @alexsherstinsky in #3748
• feat: add support for tif files by @dennisrall in #3779
• QOL: Fix slow tests GHA by @justinxzhao in #3795
• Added target_modules to LoRA config by @tgaddair in #3801
• [FEATURE] Generalize LoRA Projection Layers Tests by @alexsherstinsky in #3802
• Add support for dequantizing 4-bit bitsandbytes base models into fp16 by @arnavgarg1 in #3799
• [Maintenance] Fix deprecation warning for CharErrorRate by @arnavgarg1 in #3804
• Remove dicts from comparison metrics during visualization by @Infernaught in #3796
• Fix torch nightly tests: Temporarily add torchdata as a dependency for nightly installation by @arnavgarg1 in #3808
• Log LLM test set evaluation outputs to CSV files under the model log directory, stamped by checkpoint number. by @Infernaught in #3793
• Introduce CODEOWNERS to ensure at least 1 approval by Ludwig maintainers / the Predibase team by @arnavgarg1 in #3809
• Add utility function for uploading local folders to huggingface hub by @arnavgarg1 in #3805
• Fix LLM integration tests that fail because of PEFT 0.7 release by @arnavgarg1 in #3810
• Add AutoAugmentation to image classification training #3581 by @ethanreidel in #3798
• refactor get_total_expected_checkpoint into a function by @geoffreyangus in #3816
• Add support for IA3 PEFT Strategy by @arnavgarg1 in #3818
• Update base LLM preset list to include CodeLlama, Mistral, and Zephyr by @arnavgarg1 in #3819
• Requirements and import change (ray) by @SarthakNikhal in #3800
• Pin Transformers to < 4.36 by @arnavgarg1 in #3820
• Fix epoch computation by @geoffreyangus in #3822
• [MAINTENANCE] Support Adapter Model ".safetensors" File Format During HF Upload by @alexsherstinsky in #3823
• Move reusable LLM model methods to utility functions by @jeffkinnison in #3821
• Remove quantization config before saving dequantized weights to disk by @arnavgarg1 in #3825
• unpin transformers by @geoffreyangus in #3826
• [maintenance] small fix in log message for save_dequantized_base_model by @arnavgarg1 in #3829
• Add support for Phi-1 and Phi 1.5 by @arnavgarg1 in #3831
• hack: workaround torch bug to unblock mixtral fine-tuning by @geoffreyangus in #3830
• Add LLM Text Encoder by @jeffkinnison in #3828
• pin transformers to patch commit by @geoffreyangus in #3832
• enh: Updating requirements to pin huggingface-hub>=0.19.0 by @brightsparc in #3834
• Add support for Phi 2 by @arnavgarg1 in #3838
• Unpin transformers in favor of 4.36.2 by @arnavgarg1 in #3839
• Update LLM trainer defaults by @arnavgarg1 in #3840
• Update ludwig version to v0.9. by @justinxzhao in #3842

New Contributors

• @thelinuxkid made their first contribution in #3560
• @saad-palapa made their first contribution in #3628
• @alexsherstinsky made their first contribution in #3649
• @trevenrawr made their first contribution in #3731
• @jetaudio made their first contribution in #3750
• @ethanreidel made their first contribution in #3798
• @SarthakNikhal made their first contribution in #3800

Full Changelog: v0.8...v0.9