Coming next

The next product adds model training to Albireo, alongside model execution.

In preparation

A training extension for Albireo

The extension will let an organisation train its own models with Albireo, by fine-tuning an existing model or from scratch, in two modes.

No release date is announced. The scope may change with the needs of the first partners.

Supervised

Training runs are prepared, launched and followed from Albireo, on the data of one site.

Federated

Several computers train the same model on their own data. The updates of the model travel; the data stays where it is.

Building blocks already in the library

  • Supervised fine-tuning with prompt masking, through the chat template of each model.
  • LoRA and DoRA adapters on the attention and feed-forward paths, merged back into the weights.
  • Distillation of a teacher into a student of another architecture, from recorded logits.
  • Depth pruning of a published model, with the remaining blocks repaired by distillation.
  • V²RAM, to train a model whose working set exceeds the memory of the card.
  • Routing of the training operations to the integrated GPU of the processor.

Other items on the roadmap

QLoRA
Keeping the frozen base quantized in memory during fine-tuning.
Preference optimization
DPO with the frozen reference obtained from the same network, its adapters neutralized.
Mixture of experts on the static path
Making these models trainable and convertible like any other.
A model of the library's own
A small language model on a topology and a vocabulary chosen here.

Define this extension with us

Tell us which models you want to train and where your data resides, or support this development.