Releases: mosaicml/llm-foundry
v0.4.0
🚀 LLM Foundry v0.4.0
LLM Foundry is an efficient codebase for training, evaluating, and deploying Large Language Models (LLMs) and serves as the foundation for the MPT-7B and MPT-30B models.
In addition to the usual bug fixes and performance improvements, we've added lots of new features!
New Features
Automatic sequence packing (#683)
You can now specify packing_ratio: auto
under your finetuning dataset, to automatically profile and select a good packing ratio to efficiently pack your sequences together on the fly during finetuning. This can dramatically reduce the amount of compute wasted on padding tokens.
Flash Attention 2 (#651, #666, #672)
We now support using Flash Attention 2 both in MPT and in any model that supports Flash Attention 2 via the Transformers library. See the training instructions to learn how to use the different versions of Flash Attention.
New PyTorch, Composer, Streaming, and Transformers versions (#648, #672, #736)
As always, we've updated to new versions of the core dependencies of LLM Foundry, bringing better performance, new features, and support for new models (codellama and mistral in particular).
Easy Databricks model deployment (#618)
We've made it much easier to go from a training run to a served model using Databricks model serving. To make use of this feature, you need to specify both an MLFlowLogger
and a HuggingFaceCheckpointer
for your run.
The MLFlowLogger
should have a Unity Catalog model registry prefix in the form of catalog.schema
. This specifies where to register your models to. For example,
loggers:
mlflow:
experiment_name: /Users/[email protected]/my_experiment_name
tracking_uri: databricks
model_registry_prefix: catalog.schema
model_registry_uri: databricks-uc
The HuggingFaceCheckpointer
should specify the name you want to register the model under. For example,
callbacks:
hf_checkpointer:
save_interval: 1ep # Save Hugging Face formatted checkpoints each epoch
save_folder: s3://bucket/path/to/my/checkpoints
mlflow_registered_model_name: my_model_name # Final model will be registered to catalog.schema.my_model_name
MPT model configurations
We've added a few new options when training with the MPT architecture in LLM Foundry.
- Rotary embeddings (#675)
- (Un)Tied word embeddings (#728)
- Fine grained activation checkpointing (#720)
Evaluation Improvements
We've released v0.1 of our Eval Gauntlet (#674, #748)! This adds many new benchmarks, chain-of-thought, and a new safety category. Check out the README for full details!
In addition, we've made a few improvements to our evaluation options, with more to come!
- Allow specifying multiple evaluation datasets to compute cross entropy and perplexity on during training (#603)
- Easier versions of the HumanEval dataset, which can be useful for comparing smaller models (#645)
- More options for averaging the results of the Eval Gauntlet (#640)
New pretraining benchmarks (#543)
Added H100 profiling results to our benchmarking table.
Quality of life improvements
- Improved Generate callback with more logging options. Use the
Generate
callback to log generations from your model over the course of training. (#631) - Count number of tokens during training excluding padding tokens. Previously this count included padding tokens. (#676)
- Use the PyTorch profiler to profile your training runs. (#678)
- A convenience script for using the much faster Hugging Face
snapshot_download
to download models from the Hugging Face Hub. (#708) - New AWS specific Docker images with LLM Foundry dependencies pre-installed. (#731)
Experimental features
Inverse square root learning rate scheduler (#657)
We've added experimental support for the inverse square root learning rate scheduler.
Breaking changes
Updated Streaming defaults (#723)
We've upgraded to the latest Streaming version, including vastly improved default settings for partitioning and shuffling. This means that if you were using the defaults, you will get different results after upgrading. The new defaults should be more performant for the large majority of use cases. See the Streaming release notes for more details.
Removed support for PrefixLM for Bloom and OPT models (#704)
We occasionally remove unused experimental parts of the code base to focus on new features and better support for existing features, and we've removed support for PrefixLM applied to Bloom and OPT models in this release.
What's Changed
- Multi eval dataset logging by @snarayan21 in #603
- Merge release 0.3.0 back to main by @dakinggg in #635
- Add tmp path retention policy by @j316chuck in #641
- Add flag to disable train metrics by @mvpatel2000 in #642
- Update pins to latest version that were missed by @dakinggg in #646
- Fix overriding of rope_scaling config by @dakinggg in #644
- Add 2.1 images to docker workflow and tests by @dakinggg in #648
- Fixes to lion8b test for torch 2.1 by @dakinggg in #649
- Only log "changing autoresume" when actually changing by @aspfohl in #653
- Fix lion8b error correction with torch 2.1 by @dblalock in #656
- Clean up processes between distributed gpu tests by @j316chuck in #660
- Revert "Clean up processes between distributed gpu tests (#660)" by @j316chuck in #662
- Switch ordering of foundry gpu tests by @j316chuck in #665
- Change batch size on coding tasks to 1 to avoid OOM by @bmosaicml in #654
- Add images with flash attention 2 by @dakinggg in #651
- Fix yaml change by @dakinggg in #667
- Revert actions change by @dakinggg in #668
- Inverse Square Root LR Schedule by @mansheej in #657
- Add test suite for flash attention 2 by @dakinggg in #666
- Adding Simplified Coding Tasks by @mcarbin in #645
- Fix typo in image name by @dakinggg in #669
- Point to composer.callback.Generate by @aspfohl in #631
- Do not update past_key_values in place by @irenedea in #652
- Fix small typos in the eval readme by @maxisawesome in #671
- Convert to DataSpec and add token counts that include padding by @dakinggg in #676
- Add support for automatically registering models to UC at the end of training by @dakinggg in #618
- add
load_strict_model_weights
as an optional config parameter by @AllenHW in #655 - Small changes to HF repo update script by @dakinggg in #680
- Add profiler support in llm foundry by @j316chuck in #678
- Update_pretrain_benchmarks by @crinard in #543
- add |--...
v0.3.0
🚀 LLM Foundry v0.3.0
LLM Foundry is an efficient codebase for training, evaluating, and deploying Large Language Models (LLMs) and serves as the foundation for the MPT model series. This release includes lots of bug fixes, stability improvements, and improved error messages, in addition to all the new features listed below!
Features
Llama-2 (#485, #520, #533)
Adds support for training Llama-2 models with optimized flash attention. To enable flash attention, set the attention_patch_type
in your yaml like so:
model:
...
attention_patch_type: triton
...
See the example yaml for a full example of how to finetune Llama-2 on the MosaicML platform.
8-bit Lion (#514)
We have implemented an 8-bit version of the Lion optimizer. This reduces the memory needed per parameter from 12 bits to 9 bits. To switch from Lion to 8-bit Lion, simply change the optimizer name from decoupled_lionw
to decoupled_lionw_8b
!
Checkpoint conversion (#526, #519, #594)
We've greatly improved our utilities for checkpoint conversion, including generalizing the Composer to Hugging Face conversion script to support all causal LMs, adding a callback to perform the conversion to Hugging Face format during the training job, and support for Faster Transformer conversion from a Composer MPT checkpoint.
To enable the new callback, add the hf_checkpointer
callback to your yaml like so:
callbacks:
...
hf_checkpointer:
# Save a Hugging Face formatted checkpoint at the end of each epoch
save_interval: 1ep
# The Hugging Face formatted checkpoints will be saved inside a subfolder called huggingface,
# so this folder will likely be the same as your overall save_folder
save_folder: ./{run_name}/checkpoints
# Set the precision you want the checkpoint saved in
precision: bfloat16
Code evaluation (#587)
We have added support for running HumanEval (code evaluation) using LLM Foundry! See the evaluation readme for a more detailed description and the tasks yaml for an ICL yaml that can be used to run the HumanEval evaluation task.
Transformer Engine support (#432)
Adds support for using NVIDIA's Transformer Enginer to enable FP8 training. To enable, set fc_type='te'
and/or ffn_config['ffn_type']='te_ln_mlp'
and precision='amp_fp8'
.
MLFlow (#475)
Adds support for using MLFlow as an experiment tracker. To enable, simply add mlflow
to the loggers
section of your yaml. See the Composer docs for more configuration options for MLFlow. Stay tuned for automatic model logging to MLFlow for easy deployment.
Updated streaming version/defaults (#503, #573, #580, #602)
Updates to the latest release of MosaicML Streaming and sets better defaults for improved shuffling quality and training throughput. Check out the Streaming release notes for the full details of all the new options!
Grouped Query Attention (#492)
Implements Grouped Query Attention, which can strike a good balance between the quality of Multi Head Attention and the speed of Multi Query Attention. To enable, set attn_config['attn_type']='grouped_query_attention'
and attn_config['kv_n_heads']
to the desired number of kv heads.
MPT quality of life improvements (#559, #599)
Thanks to @tdoublep and @lorabit110 for making MPT a bit easier to use with other parts of the NLP ecosystem!
Eval gauntlet during training, inference API eval wrapper (#501, #494)
Improvements to our evaluation setup, including the ability to run the eval gauntlet during training, and a wrapper to allow using inference APIs with our eval gauntlet. The ICL tasks and gauntlet can be specified as shown [here](https://github.com/mosaicml/llm-foundry/blob/fd36398dad5ac9fde085af679514189ce9439be4/scripts/eval/yamls/hf_eval.yaml#L46-L47.
tiktoken support (#610)
We have enabled training with tiktoken tokenizers with a thin wrapper around the tiktoken library for compatibility with all the tooling built around Hugging Face tokenizers. You can enable this with a simple change to the tokenizer section of your yaml:
tokenizer:
name: tiktoken
kwargs:
model_name: gpt-4
LoRA eval (#515)
Allows the use of our evaluation script with a model trained using LoRA. Coming soon, full support for LoRA with FSDP! See this yaml for an example of evaluating a model trained using LoRA. Stay tuned for full LoRA support with FSDP!
Finetuning API
Lastly, we are building a finetuning API on top of LLM Foundry, Composer, and Streaming. Please reach out if you might be interested in using this API as a customer!
What's Changed
- Release/v0.2.0 by @vchiley in #410
- Update README.md by @abhi-mosaic in #429
- Remove try catch in eval.py; make model_gauntlet optional in eval.py by @bmosaicml in #434
- Use torch.repeat instead of expand on key & value in Triton MQA to prevent NaNs with certain h_dims by @sashaDoubov in #442
- Update mcli-hf-generate.yaml by @vchiley in #456
- Add trust remote code for tokenizer in inference conversion script by @margaretqian in #446
- setup.py: replace composer with mosaicml by @guoyejun in #458
- Add linear layer and ffn config to enable TransformerEngine layers (with FP8) by @vchiley in #432
- use mono checkpoints by @samhavens in #448
- Update inference benchmark with recent HF changes by @sashaDoubov in #461
- Adding different device intialization to eval by @bcui19 in #466
- Fix missing import by @bcui19 in #470
- Autoresume default on by @mvpatel2000 in #467
- Support eval loader when finetuning from JSONL files in object stores by @samhavens in #469
- Fix ambiguous
throughput
in README by @abhi-mosaic in #476 - adds early stopping call back by @codestar12 in #488
- Update accelerate to 0.20.3 for LLaMa-2 support by @rishab-partha in #485
- If Alibi is on, we should turn learned_pos_emb to False by @bcui19 in #489
- Fix Local World Size by @rishab-partha in #495
- Increase lint CI timeout by @dakinggg in #498
- fix boolean for reentrant setting by @dakinggg in #500
- Adding pyright to pre-commit by @bcui19 in #477
- fix no bias assignment by @vchiley in #502
- Updated StreamingTextDataset to pass take in shuffle_block_size by @snarayan21 in #503
- Add MLFlow as a logger option by @aspfohl in #475
- Remove old optimizer logs by @mvpatel2000 in #509
- Updates GPU test timeout to use mcloud flag by @mvpatel2000 in #510
- Grouped Query Attention + Refactor Attn by @sashaDoubov in #492
- Fix training integration test by @j316chuck in #517
- Update max duration due to mcli api change by @mvpatel2000 in #523
- Fix typo in GQA comments by @sashaDoubov in https://github.com/mosaic...
v0.2.0
🚀 LLM Foundry v0.2.0
LLM Foundry is an efficient codebase for training, evaluating, and deploying Large Language Models (LLM). LLM Foundry serves as the efficient training codebase for the MPT-7B and MPT-30B models. Our emphasis is on efficiency, scalability, and ease-of-use, to enable fast iteration and prototyping.
We are excited to share the release of v0.2.0
, packed with support for new hardware, features, and tutorials.
📖 Tutorials
We have released new tutorial content and helper scripts for dataset preparation, pre-training, fine-tuning, and inference!
To start off, a basic walkthrough and answers to FAQs can be found in our Basic Tutorial.
Next, detailed guides for different workflows are linked below:
Training
In addition, for a more advanced and self-contained example of finetuning the MPT-7B model, see Finetune Example.
Inference
The inference tutorials cover several new features we've added that improve integration with HuggingFace and FasterTransformer libraries:
- Converting a Composer checkpoint to an HF checkpoint folder
- Interactive Generation with HF models
- Interactive Chat with HF models
- Converting an HF model to ONNX
- Converting an HF MPT to FasterTransformer
- Running MPT with FasterTransformer
Major Features
LLM Foundry now uses Composer v0.15.0
and Streaming v0.5.1
as minimum requirements. For more details, see their release notes for Composer and Streaming for all the improvements.
-
🆕 Torch 2.0 support
LLM Foundry is now Torch 2.0 compatible!
Note: we have not tested
torch.compile
, but do not expect significant performance improvements. -
⚡ H100 Support
We now support NVIDIA H100 systems! See our blog post on Benchmarking LLMs on H100 GPUs for initial performance and convergence details.
To run LLM Foundry with NVIDIA H100 systems, be sure to use a docker images that has CUDA 11.8+ and PyTorch 2.0+ versions.
For example,
mosaicml/pytorch:2.0.1_cu118-python3.10-ubuntu20.04
from our dockerhub has been tested with NVIDIA H100 systems.No code changes should be required.
-
📈 AMD MI250 GPU Support
With the release of PyTorch 2.0 and ROCm 5.4+, excited to share that LLM training now works out of the box on AMD Datacenter GPUs! Read our blog post on Training LLMs with AMD MI250 GPUs for more details.
Running with our stack was straightforward: use the ROCm 5.4 docker image
rocm/dev-ubuntu-20.04:5.4.3-complete
; and install PyTorch for ROCm 5.4 and install Flash Attention.Modify your configuration settings:
attn_impl=flash
instead of the defaulttriton
- Note: ALiBi is currently not supported with
attn_impl=flash
.
- Note: ALiBi is currently not supported with
loss_fn=torch_crossentropy
instead of the defaultfused_crossentropy
.
-
🚧 LoRA finetuning (Preview)
We have included a preview release of Low Rank Adaptation (LoRA) support for memory-efficient fine-tuning of LLMs (Shen et al, 2021).
To use LoRA, follow the instructions found here.
Note: This is a preview feature, please let us know any feedback! The API and support is subject to change.
-
🔎 Evaluation Refactor (#308)
Our evaluation suite has been significantly refactored into our Model Gauntlet approach. This includes a number of breaking API changes to support multiple models:
- Instead of
model
, use themodels
keyword and provide a list of models. tokenizer
is now model-specific.
For example, to run the gauntlet of various eval tasks with
mosaicml/mpt-7b
:cd llm-foundry/scripts composer eval/eval.py eval/yamls/hf_eval.yaml model_name_or_path=mosaicml/mpt-7b
This release also makes evaluation deterministic even on different number of GPUs.
For more details on all these changes, see #308
- Instead of
-
⏱️ Benchmarking Inference
To better support the deployment of LLMs, we have included inference benchmarking suite and results across different hardware and other LLM models.
PR List
- hf dict cfg overrides by @vchiley in #90
- Add slack and license buttons to readme by @growlix in #98
- Add minimum
mosaicml-streaming
version by @hanlint in #110 - Update dataloader.py by @nelsontkq in #102
- Add features to hf_generate by @alextrott16 in #116
- Make mpt7b finetuning more obvious by @samhavens in #101
- Fix(finetune yaml): fix parameters in mpt-7b_dolly_sft.yaml by @alanxmay in #131
- Fix HF conversion script to upload to S3 after editing the files to be HF compatible by @dakinggg in #136
- Set pad_token_id to tokenizer.pad_token_id if not set on command line by @patrickhwood in #118
- Changed the keep_zip default to False to comply with StreamingDataset by @karan6181 in #150
- Add cloud upload to checkpoint conversion script by @dakinggg in #151
- Adds precision to eval by @mvpatel2000 in #148
- Update StreamingDataset defaults by @abhi-mosaic in #157
- Explain
composer
command by @hanlint in #164 - Remove
pynvml
by @hanlint in #165 - Adds a concrete finetuning example from a custom dataset by @alextrott16 in #156
- Remove health checker by @mvpatel2000 in #167
- Rename datasets to avoid hf conflict by @hanlint in #175
- Torch2 (#177) by @vchiley in #178
- Revert "Torch2 (#177) (#178)" by @dakinggg in #181
- clean up dataset conversion readme by @codestar12 in #168
- Convert MPT checkpoints to FT format by @dskhudia in #169
- Update README.md by @jacobfulano in #198
- Removed unused
tokenizer_name
config field by @dakinggg in #206 - Add community links to README by @hanlint in #182
- Add Tensorboard logger to yaml config by @hanlint in #166
- Update inference README by @abhi-mosaic in #204
- torch2 updt with hf fixes by @vchiley in #193
- Removing deprecated vocabulary size parameter from composer CE metrics by @sashaDoubov in #222
- Add `composer[...
v0.1.1
What's New
LLM Foundry is now on PyPI!
What's Changed
- Update README.md by @ejyuen in #72
- Update version by @dakinggg in #73
- Remove todo in workflow by @mvpatel2000 in #74
- Bump composer version by @vchiley in #84
- Fix pypi by @mvpatel2000 in #80
- Remove xentropy from pypi by @mvpatel2000 in #86
- Fix sed command for xentropy by @mvpatel2000 in #87
- Updates to prefixlm and t5 by @alextrott16 in #85
- Disable image for pypi by @mvpatel2000 in #97
New Contributors
Full Changelog: v0.1.0...v0.1.1
Announcing LLM Foundry and the MPT foundation series
🚀 LLM Foundry v0.1.0
This is the first release of MosaicML's LLM Foundry!
Our efficient code for training, evaluating, and deploying LLMs outgrew our examples repository, so we've migrated to a brand new repository dedicated to everything LLMs. Keep watching this space and see the top-level README and our blog post for more details on this announcement!
Model releases
In addition to all the open-source code released here, we're releasing four open-source models that we hope will be useful to the community. All models were trained on the MosaicML platform, using Composer and Streaming. If you're interested in training your own models, or using these models with our optimized inference stack, please reach out!
mpt-7b
: This is our base 7-billion parameter model, trained for 1 trillion tokens. This model is released with an Apache-2.0 (commercial use permitted) license.mpt-7b-storywriter
: All of the models use ALiBi to allow them to exrapolate to longer sequence lengths than they saw during training, but storywriter is our long context model, further pretrained on 65k-token excerpts of a fiction subset of the books3 corpus. This model is released with an Apache-2.0 (commercial use permitted) license.mpt-7b-instruct
: This model is instruction finetuned on a dataset we also release, derived from Databrick's Dolly-15k and Anthropic’s Helpful and Harmless datasets. This model is released with a CC-By-SA-3.0 (commercial use permitted) license.mpt-7b-chat
: This model is trained to be able to chat by further training on the ShareGPT-Vicuna, HC3, Alpaca, Helpful and Harmless, and Evol-Instruct datasets. This model is released with a CC-By-NC-SA-4.0 (non-commercial use only) license.
Features
Training
We release fully featured code for efficiently training any HuggingFace LLM (including our optimized MPT using FSDP, Composer, and Streaming. Seamlessly scale to multi-gpu and multi-node training, stream your data from one cloud, train on a different cloud, write checkpoints to a third cloud, send your training logs to Weights&Biases, and much more. See the README for more detailed instructions on getting started pretraining and finetuning!
Our MPT model is equipped with the latest advancements in training large transformers (e.g. ALiBi, the LION optimizer, FlashAttention), and is desgined to be easily hackable, configurable, and extendable!
Evaluation
Our evaluation framework, makes it easy to fully re-evaluate any HuggingFace model. We also include copies of the processed data for many popular benchmarks, to make it easy to replicate our evals, and perform your own! We welcome the addition of new benchmarks to our suite. In previous benchmarks, our setup is 8x faster than other eval frameworks on a single GPU and seamlessly achieves linear scaling with multiple GPUs. Built-in support for FSDP makes it possible to evaluate large models and use larger batch sizes for further acceleration.
Inference
MPT is designed to be fast, easy, and cheap to deploy for inference. To begin with, all MPT models are subclassed from the HuggingFace PretrainedModel base class, which means that they are fully compatible with the HuggingFace ecosystem. You can upload MPT models to the HuggingFace Hub, generate outputs with standard pipelines like model.generate(...)
, build HuggingFace Spaces (see some of ours here!), and more.
What about performance? With MPT’s optimized layers (including FlashAttention and low precision layernorm), the out-of-the-box performance of MPT-7B on GPUs when using model.generate(...)
is 1.5x-2x faster than other 7B models like LLaMa-7B. This makes it easy to build fast and flexible inference pipelines with just HuggingFace and PyTorch.
Finally, for the best hosting experience, deploy your MPT models directly on MosaicML’s Inference service. Start with our managed endpoints for models like MPT-7B-Instruct, and/or deploy your own custom model endpoints for optimal cost and data privacy. Check out the Inference blog post for more details!