Dlib-Transformer-extensions
The library on which everything we publish rests. This temporary fork of Dlib, the open-source C++ machine learning library, is the place where each mechanism is designed, demonstrated by a runnable example, measured and stabilised before it is proposed to Dlib itself.
- C++14
- Boost Software License 1.0
- No Python in the chain
- CUDA and OpenVINO
- 24
- example programs, one per mechanism
- 14
- contributions already merged into Dlib
- 12
- model architectures read from GGUF files
- 0
- Python dependency in the chain
A temporary fork, by design
Dlib moves at the pace of a reference library: each addition must be general, tested and documented. Language models move faster. The fork gives each mechanism room to mature, then returns it upstream: fourteen contributions have already been merged into Dlib, and the products are built on what the fork has stabilised.
Cygnus, the target architecture
Cygnus is the family of small language models built on the architecture the library aims at: grouped query attention with rotary position encoding and a key-value cache, and a mixture-of-experts feed-forward sublayer with sparse routing. At inference, only a fraction of the parameters works for each token.
Open-weight models, imported and verified
The import reads a published model in six stages and produces a Dlib network type with its tokenizer. Every position of a test prompt is then compared with the dynamic runtime and with an external reference implementation: an imported model computes what its publisher released.
Memory and devices, managed below the network
Two mechanisms act under the layers, with no change to a network or to a training loop. V²RAM decides where each block of tensors lives, under a budget smaller than the model. Device routing times each operation on the host, on the NPU and on the integrated GPU, and keeps the fastest.
24 examples, one per mechanism
Each mechanism of the library comes with a program that builds it, runs it and validates it: documentation that always stays up to date.
First models
A character-level model, then the canonical transformer, trained until it reproduces its corpus byte for byte.
slm_basic_train_ex slm_advanced_train_ex slm_enwiki_train_ex
Attention and feed-forward variants
Grouped query attention, adaptive computation time, the fused attention layer with its key-value cache, mixture of experts.
slm_advanced_gqa_train_ex slm_advanced_gqa_kvc_train_ex slm_transformer_configs_ex
The Cygnus series
Foundation pre-training on a corpus, then instruction tuning and an interactive chat.
slm_cygnus_foundation_ex slm_cygnus_instruct_ex
Open-weight models
GGUF import and dynamic runtime, LoRA and DoRA fine-tuning with its numerical validation, distillation, embeddings and document search.
slm_gguf_import_ex slm_gguf_runtime_ex slm_lora_finetune_ex slm_lora_adapter_check_ex slm_distill_ex slm_embed_ex
Vision
The vision tower of a multimodal model, a classifier trained from scratch on CIFAR-10, self-supervised training with Barlow Twins.
slm_vision_tower_ex slm_vit_classify_ex slm_vit_ssl_ex
Memory and devices
A network larger than the graphics card, and operations routed to the NPU and to the integrated GPU.
slm_extended_memory_ex slm_npu_offload_ex
Reasoning and applications
Hierarchical reasoning on Sudoku-Extreme and ARC-AGI, document anonymization, lossless compression by prediction.
slm_hrm_sudoku_ex slm_hrm_arc_puzzle_ex slm_document_anonymizer_ex slm_predictive_compressor_ex slm_tok_predictive_compressor_ex
Work in progress to extend the capabilities of the library
- QLoRA
- Keeping the frozen base quantized in memory during fine-tuning.
- Preference optimization
- DPO with the frozen reference obtained from the same network, its adapters neutralized.
- Mixture of experts on the static path
- Making these models trainable and convertible like any other.
- A model of the library's own
- A small language model on a topology and a vocabulary chosen here.
Use the library, or contribute to its development
The code is public under a permissive licence. Contributions, sponsorship and partnerships all move the library forward.





