Kolibri: Aleph Alpha's open German-English MoE model with 78B parameters

What this is about

Kolibri is an English-German language model that, according to its technical report, is being released as open weights under the Apache 2.0 license. Per alphaXiv's AI summary, Aleph Alpha is behind the report. The model is built for "sovereign" and specialized use, with a focus on German and regulated sectors such as public administration, industry and aviation. The authors stress that they control the entire development process: data, architecture, training infrastructure, post-training and evaluation. The sources don't give a link to the weights, so a Hugging Face repo is unknown.

Architecture: lots of capacity, little compute

Kolibri is a mixture-of-experts (MoE) model: each token passes only through selected expert networks instead of all parameters. In total there are 78.1B parameters (roughly, the knobs learned during training), but only 3.46B are active per token, or 4.4%. Each MoE layer picks six of 384 routed experts and also uses one shared expert.

A controlled sparsity test held active parameters at about 3.4B and varied the total size. Going from 42.1B to 82.3B, all quality metrics improved. From 82.3B to 122.6B, they improved only slightly. In decoding, the 82.3B and 122.6B variants were 13% and 32% slower than the smallest at 4k context, and 4% and 12% slower at 256k context. On two H100 GPUs, the 122.6B variant fit three concurrent requests with 256k tokens, while the 82.3B variant fit 18. The authors therefore aimed for around 80B total parameters.

The second building block is hybrid attention: 10 blocks with full attention and 40 with sliding-window attention over the preceding 512 tokens. In a proxy comparison at equal training compute, the authors say long-context scores improved with no loss on pre-training benchmarks. However, the compared models also differed in training length and positional encoding.

Training

The model was trained on 24 trillion tokens across pre-training, mid-training and context extension. German accounts for more than 20%. To reach that share, more than 2 trillion German tokens were added to existing German datasets, drawn from a custom web curation pipeline and synthetic generation. Post-training combines supervised fine-tuning on curated German and English data with reinforcement learning on over 1.2 million internally curated tasks (reasoning, tool use, instruction following, code, retrieval).

Kolibri builds on Kolibri Origin, an earlier iteration that validated the pipeline at scale. Both come from the "Model Factory," which links data, experiments, training, evaluation and deployment through versioned workflows. This is meant to ensure traceability and repeatability, and it was designed with regulation in mind. According to alphaXiv, though, it comes with no certification or guarantee for the behavior of downstream systems.

Results

"Overall" is the unweighted mean of the category averages. English averages eight categories, and German averages the four categories with German benchmarks. Throughput was measured on eight B200 GPUs at 16k context, in UTF-8 bytes per second per GPU, with FP8 weights and KV cache. The baselines' quality scores mostly come from BF16 checkpoints. This is a proxy for high concurrency, not a measure of latency, total cost or energy use.

Among the open models tested, Kolibri sits on the Pareto frontier: no model offers higher quality at the same throughput or higher throughput at the same quality. Compared with Qwen3.5 35B-A3B (3B active), Kolibri scores 0.8 points higher in English and 1.0 in German. Both sit on the frontier. The dense 27B Qwen3.8 scores higher overall, but activates almost eight times as many parameters.

What you need to try it

The sources list no requirements for end users. As a rough estimate, 78.1B parameters need about 45–50 GB of memory at 4-bit quantization. That means several consumer GPUs or lots of system RAM with offloading. With only 3.46B active parameters, generation is relatively fast, provided the weights sit in RAM. The report's benchmarks ran on B200 and H100 servers.

Our take

Kolibri is a rare open model with an explicit German focus under a permissive license. That matters for public agencies and industry with privacy and sovereignty requirements. The efficient MoE design targets running costs. Some caveats remain: all figures come from the report itself, and the gain over Kolibri Origin has no single cause, since many stages changed at once. The Pareto claim holds only for the selected models and the chosen measurement setup, and stronger dense models like Qwen3.8 score better overall. Whether the model delivers in practice is for independent tests to show.