Colibri: Run 744B AI Models on Your Potato PC — How It Actually Works

Colibrì at a glance

Why a 744-billion-parameter model can run on a laptop at all

The trick is not compression and not a smaller model. It is the shape of the model. GLM-5.2 is a Mixture-of-Experts (MoE) model: instead of one giant network, it is split into many small sub-networks called experts. For every token it generates, a router picks a handful of experts and ignores all the rest.

The numbers for GLM-5.2, as documented by the project and independent write-ups:

• 744 billion parameters in total (Hugging Face lists 753.3 billion — the 9.66 billion difference is the multi-token prediction layer, which Colibrì excludes from its count).
• ~40 billion parameters active per token — roughly 5–7% of the model. The rest sits idle for that token.
• 78 layers, 75 of them MoE layers, with 256 experts per layer plus one shared expert. The router selects 8 experts per layer, per token.
• 19,456 experts in total, each about 19 MB at int4 — together roughly 370 GB.
• Per token, about 600 experts are touched — around 11 GB of expert weights out of the 370 GB total.

Most inference engines ignore this sparsity and load the whole model into fast memory anyway. Colibrì's README states the alternative in one line: the model does not have to fit in fast memory; it has to be placed. The problem changes from "fit 400 GB in memory" to "move ~11 GB per token from SSD to CPU, minus whatever is already cached". That is still a lot of I/O — and it is the whole story of Colibrì's speed.

How it actually works: RAM vs SSD, layer by layer

Colibrì treats VRAM, RAM and NVMe as three placement tiers for the same weights, not as separate worlds. The engine splits the model by how often each piece gets touched:

• VRAM (graphics card memory) — the hottest experts. If a GPU is present, the most frequently used experts live here. No GPU is required; this tier is simply empty on a CPU-only machine.
• RAM — the dense core, permanently resident. The parts every token needs — attention, the shared expert, embeddings, about 17 billion parameters — stay in RAM the whole time. At int4 precision that is roughly 9.9–10 GB. RAM also holds a cache of "warm" experts that have been used recently.
• NVMe SSD — everything else. The ~370 GB of routed experts live on disk. When the router selects an expert that is not cached, Colibrì reads it from the SSD, uses it for that token, and then either keeps it in the cache or drops it.

Each expert's three weight matrices are stored next to each other on disk, so the engine pulls an expert in a single pread (one read of a chunk of a file at a given offset) instead of three scattered reads. A loader pool reads the experts that are missing from memory while the experts already resident are being computed on — loading and computing overlap instead of taking turns. The project describes its per-token pipeline in five steps: route, union, place, overlap, learn.

The "learn" step is a file called .coli_usage: it records which experts your actual conversations keep activating, and pins that set into fast memory. The more you use the model, the better its cache matches your usage — which is why the project distinguishes a "cold" number (25 GB laptop, nothing cached) from a "warm" one (a 128 GB desktop with a learned cache). Expert files can also be spread across several NVMe drives with weighted read distribution; the project reports a 37.5% decode improvement from adding a second drive.

RAM consumption, explained — does the "99% on SSD" claim check out?

Verdict: essentially yes — with one correction. The user heard that only the active task stays in RAM and ~99% stays on SSD. The verified split is: about 370 GB of 380 GB of weights (~97%) live on the SSD, and roughly 10 GB — the dense core — is permanently resident in RAM, plus a cache of recently used experts. So it is not true that only the currently active layer sits in RAM: the dense core never leaves RAM, because every single token needs it. What is true is that the overwhelming majority of the model — the routed experts — never enters RAM until the router asks for it, and most of it never enters RAM at all during a given session.

Two consequences follow. First, RAM is no longer the wall — disk bandwidth is. Every token needs ~11 GB of expert weights moved across the storage bus, so a slow SATA SSD or a nearly full drive will crawl, while a fast PCIe NVMe drive is the single best upgrade. Second, the cache matters enormously: in the project's own OLMoE test, cutting the expert cache from 64 to 8 experts per layer dropped the median speed from 2.84 to 1.52 tokens/sec — and with disk reads capped at 300 MiB/s, from 1.25 to 0.35 tokens/sec. More RAM does not make the model bigger; it makes the cache bigger, and a bigger cache means fewer disk reads.

Speed: what the numbers actually say

Colibrì's own README benchmark table is unusually honest, and it is the source for every number below. The headline figure and the catch live in the same table:

• GLM-5.2 on the 25 GB laptop: 0.05–0.1 tokens per second. That is 10 to 20 seconds per token. One hundred output tokens take roughly 16–33 minutes. It runs; it is not interactive.
• GLM-5.2 on six RTX 5090s: 6.8 tokens per second. Same model, same engine, server-class hardware — about 70–130× faster than the laptop.
• Qwen3.6-35B-A3B on one RTX PRO 4000: 104.8 tokens/sec with CUDA, 92.9 with Vulkan, 13.9 on the processor alone. The same card, the same smaller model, three backends — the gap between GPU and CPU is the story.
• Prompt processing is a different world: on that same card, Colibrì processes your prompt at 3,491 tokens per second. Reading the prompt (prefill) can be parallelised; generating the reply (decode) cannot — each token depends on the one before it.

Output quality: same weights, same answer — speed is what changes

This is the question that matters most, and the answer is reassuring: where a weight lives does not change what it computes. Placing an expert on disk or in VRAM changes the speed, not the weights' precision or the router's decisions. Given the same weights, the same precision and the same sampling settings, Colibrì produces the same tokens as a server running the same model — the project phrases its own guarantee as "no SLA on speed, and a hard guarantee on semantics".

There is one honest caveat, and it is about precision, not placement. Colibrì runs GLM-5.2 at int4 (4-bit) quantization — that is how 744 billion parameters fit into a 429 GB file. A datacenter deployment of the same model often runs at higher precision (BF16 or FP8), and a lower-precision model can give slightly different — usually slightly worse — answers on some tasks. So if you compare Colibrì's int4 GLM-5.2 against a server's BF16 GLM-5.2, small quality differences are possible. But those come from the quantization, not from the SSD streaming. Against the same int4 weights on a server, the output is identical; it just arrives later.

In practice that means: for research, analysis, code review, long-form drafting and anything where you can wait, the laptop's answer is the frontier model's answer. For live chat, it is not — a 20-second wait per word breaks any conversation.

Laptop vs server: how long for the same output?

Using only the project's own benchmark rates (GLM-5.2: 0.05–0.1 tok/s on the 25 GB laptop; 6.8 tok/s on six RTX 5090s), here is what the same reply costs in time on each. The times below are calculated from those published rates, not separately measured:

For scale, a smaller model on a single workstation card sits between the two: Qwen3.6-35B-A3B on one RTX PRO 4000 decodes at 104.8 tok/s with CUDA — a 700-token reply in about 7 seconds, or about 50 seconds on the CPU alone (13.9 tok/s). The laptop's role is therefore clear: it is the machine you leave running overnight on a hard problem, not the machine you chat with. Colibrì itself is built for exactly that — asynchronous, batch-style work where the model is huge and the deadline is tomorrow.

Parallel processing: what Colibrì can and cannot parallelise

The user is right that Colibrì lacks parallel processing in the sense that matters most — but the reason is deeper than the engine. Token generation is inherently sequential: each new token depends on all the tokens before it, so no engine — on a laptop or a server — can generate one reply's tokens in parallel. What can be parallelised is everything around that sequence, and here Colibrì does parallelise:

• Prompt processing (prefill) is parallel: the whole prompt is processed at once, at 3,491 tokens/sec on the RTX PRO 4000 in the project's benchmarks. Your long prompt is not the slow part.
• Loading overlaps computing: the loader pool fetches the next experts from disk while the current ones are being computed on, so the disk and the CPU work at the same time instead of taking turns.
• Multiple NVMe drives stripe the reads: expert files can be spread across drives with weighted distribution, and the project reports a 37.5% decode improvement from a second drive.

What cannot be parallelised is the decode loop itself — one token, then the next, each waiting on ~11 GB of expert weights to arrive from disk. That is why the laptop is slow, and why adding CPU cores does not fix it: the bottleneck is SSD bandwidth and latency, not compute. A faster NVMe drive, a bigger expert cache (more RAM), and a second drive are the only levers that move the number.

Benefits: why anyone would accept 0.05 tokens per second

• Privacy, fully offline. Once downloaded, nothing touches the internet. Your prompts, documents and code never leave your machine — the model runs locally, behind your own firewall.
• Frontier-size models without frontier hardware. A 744B model used to mean a datacenter. Now it means a laptop, 429 GB of disk and patience. The same engine also runs Kimi K3 (2.8 trillion parameters) and Inkling (975B) — sizes no consumer GPU could ever hold.
• Zero GPU cost. No $2,000+ graphics card, no cloud GPU rental by the hour, no API bill per token. The cost is disk space and time.
• Same weights, same answers. This is not a distilled or shrunken model. It is GLM-5.2 itself, with the same capabilities — reasoning, coding, long context — at int4 precision.
• Open and hackable. Apache 2.0, pure C with no external dependencies in the engine, an OpenAI-compatible API on localhost, and a chat dashboard out of the box.
• A real alternative exists for comparison: DwarfStar (ds4), built by Redis creator Salvatore Sanfilippo, uses the same disk-streaming idea for DeepSeek V4 and is reported at ~35–40 tokens/sec on a MacBook Pro M5 Max — much faster, on a different model, on Apple Silicon. The idea is bigger than one project.

Honest limitations — read this before downloading 429 GB

• Extremely slow on the headline configuration. 10–20 seconds per token on the 25 GB laptop. It suits overnight batch jobs, not conversation.
• The download is enormous. GLM-5.2 in Colibrì's format is a ~429 GB file. Beginners should start with a smaller model (the smallest needs 22 GB free) instead.
• Very heavy SSD read traffic. Around 11 GB of expert weights cross the storage bus per token. A fast NVMe drive is effectively required; a SATA SSD or a nearly full drive will be far slower, and the cache tests show how quickly speed collapses when the disk cannot keep up.
• Young project, moving fast. Started July 2026; eight security advisories were published in August 2026; most merges go through the author. Do not expose it to the internet without understanding what you are running.
• Colibrì-only weight containers. The model files are in the project's own format — you cannot just point it at any GGUF file and go; conversion and download are part of the setup.
• Kimi K3 is text-only in Colibrì — its vision tower is not loaded. Some model features do not carry over.
• Vulkan performance was measured on exactly one graphics card in the project's own notes — expect variance on other hardware.
• It is not a ChatGPT replacement. Colibrì is an inference engine, not a product: no hosted service, no safety tuning of its own, and the quality of the answer depends on the model you choose, not on Colibrì.

How to try it (start small)

Do not start with GLM-5.2. Start with the smallest model, confirm the engine works on your machine, and only then commit to a 429 GB download.

git clone https://github.com/JustVugg/colibri

Full installation instructions are in the project's quickstart guide: github.com/JustVugg/colibri/blob/main/docs/quickstart.md. Prebuilt archives exist for Linux, macOS and Windows, or build from source with GCC/Clang plus OpenMP. The setup script inspects your RAM, disk and GPU, recommends a model that fits, downloads it, and opens a chat dashboard with an OpenAI-compatible API on localhost. To see where a model would be placed across your VRAM, RAM and disk before running it:

./coli plan --model /nvme/qwen36

Practical tips from the project's own documentation and coverage: put the model on your fastest NVMe drive (a second drive adds a reported 37.5% decode speed); give the machine as much RAM as you can, because RAM becomes expert cache; and let .coli_usage learn your usage over a few sessions before judging the speed — the cold number and the warm number are very different.

Frequently asked questions

What is Colibrì?

A free, open-source inference engine (pure C, Apache 2.0) by Vincenzo Fornaro (JustVugg) that runs huge Mixture-of-Experts AI models on ordinary computers by streaming most of the weights from an SSD. It runs GLM-5.2 (744B parameters) on a laptop with ~25 GB of RAM and no GPU.

How does Colibrì fit a 744B model in 25 GB of RAM?

It doesn't fit it — it places it. The ~10 GB dense core stays in RAM permanently; the ~370 GB of routed experts stay on the NVMe SSD and are streamed in only when the router selects them (about 11 GB per token). Only 5–7% of the model's parameters are active for any one token.

Is the output quality the same as on a server?

Yes, for the same weights at the same precision: placement changes speed, not the math. Colibrì runs GLM-5.2 at int4, so a server running the same model at higher precision (BF16/FP8) could give slightly different answers — but that is a quantization difference, not a streaming one.

How slow is it, really?

On the 25 GB laptop, GLM-5.2 runs at 0.05–0.1 tokens per second — 10–20 seconds per word, 16–33 minutes for 100 tokens. On six RTX 5090s the same model does 6.8 tokens/sec. It is for overnight batch work, not live chat.

Do I need a GPU to run Colibrì?

No. Its CPU-based inference path runs without one. A GPU helps a lot — the same engine does 104.8 tokens/sec on one RTX PRO 4000 with a smaller model (Qwen3.6-35B) versus 13.9 on the CPU alone — but it is optional.

Can it run other models besides GLM-5.2?

Yes — nine families, including DeepSeek V4/V4.1 Flash, Kimi K3 (2.8 trillion parameters, text-only in Colibrì), Qwen3.6 and Qwen3.8-Flash-Next, Inkling (975B), GLM-5.3/5.3-Flash and OLMoE, plus an image model.

Sources

• GitHub — JustVugg/colibri (README, benchmarks, quickstart)
• WISEVERSITY.IN — Colibri: Run 744B AI Models on your Potato PC without an expensive GPU
• Medium (Raghu Juluri) — Colibrì Is Not an AI Project. It's a Storage Engine. (benchmark table, 0.05–0.1 tok/s, 104.8 tok/s, 3,491 tok/s prefill, second-drive +37.5%)
• Medium — Running a 744B-Parameter Model From Your SSD: A Technical Look at Colibrì
• LocalClaw — Colibri Runs GLM-5.2 on 25GB RAM: How It Works (9.9 GB dense core, 372 GB container, Kimi K3 text-only)
• jacar.es — Colibri: giant MoE models streamed from disk (expert arithmetic, 6.8 tok/s on six RTX 5090s, OLMoE cache tests, security advisories)
• MrTechKing — Colibri: Giant AI Models on Your PC, Scanned (v2.1.0, 47,000 stars, 221 contributors, 8 GB minimum)
• Secure in Seconds — Run a 744-Billion-Parameter AI Model Locally, Off Your Own SSD (DwarfStar comparison)
• Pasquale Pillitteri — Run 744B MoE models on a laptop with disk streaming, no GPU needed

Related reading: Gemini API CLI: Complete Install and Command Guide · Gemini Omni 1.2 Flash: Google's 15-Second AI Video Model · TurboFieldfare: run Gemma 4 26B on an 8GB Mac — complete install guide