Hugging Face released Transformers 5.22.0 on October 29, 2026, adding adapter-training support for compatible FP8-compressed models. For machine-learning teams tailoring models to their own tasks, this opens a customization option that keeps the compressed base model fixed.
Adapters are small trainable additions that customize a model while leaving its base weights unchanged. Those weights are the numerical parameters learned during training. FP8 represents values in eight bits, reducing the storage needed for weights compared with higher-precision formats.
The missing piece was the learning signal. According to the adapter-support change, affected FP8 operations could previously drop gradients—the signals used to adjust trainable parameters—or fail during training. The release adds those calculations for supported dense and grouped operations; it does not turn the compressed base weights themselves into trainable parameters.
Which setups qualify
Support depends on the saved model’s compression format and its execution backend, the software path that performs its calculations:
Model format: frozen fine-grained FP8 weights are supported. Packed FP4 weights in this integration do not support the required backward calculation.
Expert backend: for mixture-of-experts models, which route work among model components, the supported paths include eager and grouped_mm. The batched_mm and deepgemm expert backends still lack backward support.
Activation settings: grouped_mm requires dynamic rather than static activation quantization—a setting governing compression of intermediate values. DeepGEMM acceleration inside a dense layer is separate from the unsupported deepgemm expert backend.
This is an extension of one training route, not the first way to customize compressed models. Hugging Face already documents four-bit adapter training with QLoRA; the FP4 exclusion above does not rule out those separate approaches.
Version 5.22.0 is available on PyPI and requires Python 3.11 or later. Hugging Face’s FP8 guide calls for a GPU with compute capability 9 or above, such as NVIDIA’s H100, plus compatible PyTorch/CUDA and Accelerate.
The released loader uses a lower CUDA eligibility threshold of 8.9. That difference matters: passing a loading check does not establish that a complete adapter-training job works on a particular device. The cited evidence does not establish a universal minimum GPU-memory requirement.
A smaller temporary allocation is not a smaller training bill
Maintainer Arthur Zucker reports that a follow-up change reduced the largest temporary memory allocation from 28.9 GiB to 2.8 GiB in an H200 layer-backward benchmark. It used DeepSeek-V3-shaped mixture-of-experts layers, 8,000 tokens and eight selected experts per token. The comparison is against the initial adapter-support implementation, not a complete training job on an older stable release.
In the same report, a separate Qwen3-30B-A3B-FP8 forward-and-backward comparison at sequence length 2,048 showed unchanged peak memory, recorded as +14.8 GiB across the compared variants. The maintainer says activations dominated that result. That figure is not the model’s total memory requirement, and neither comparison establishes a reduction in cloud spending.
The practical implication is to check the exact model format, backend and GPU setup before choosing this route. A checkpoint that generates text successfully is not necessarily ready for training. Teams should assess memory use and adapter quality on their own workload before budgeting around the change.
For the separate decision about running downloadable models rather than using hosted access, see our Mistral Large 4 self-hosting coverage. That release does not establish compatibility with this adapter-training path.