Encoder, experts and memory

There is a moment that keeps coming back in calls these last few months, and it always lands at roughly the same point. The use case has been discussed, everyone agrees an LLM is needed, and someone — usually not the engineer, usually the person who signs — says the line: «I read that these Chinese models are as good as ChatGPT and they are free. Why don't we just put one in-house and stop worrying about it?»

It is a good question. It is also a question with no one-line answer, and how you handle it in the next twenty seconds decides whether, for the rest of the project, you are the consultant who reasons or the one defending a vendor.

Answering it well takes more than knowing DeepSeek shipped a new version. It takes knowing what they changed, because that change — not the version number — is what tells you whether the model can run in the client's datacenter. What follows is the minimum needed to hold that conversation as an equal: encoder and decoder, mixture of experts, lookup memory, and what you can actually serve it with.

What actually happened, in order

Let me recap the facts, because the thread is easy to lose.

On 14 August Zhipu released GLM-5.3, presented as the strongest open model for coding. The interesting part is that it shares its base model with GLM-5.2: the entire gain comes from post-training alone. On their internal coding benchmark they claim a 50% relative jump, and on public agentic benchmarks the numbers go from 4.6 to 28.3 on Terminal-Bench 3.0 and from 46.2 to 66.9 on DeepSWE v1.1. The weights came out two weeks after the announcement, once the safety checks were done.

Qwen3.8-Flash-Next took into production an idea that until then had mostly been a paper: pairing the network with an enormous table of n-gram embeddings — 51 billion parameters, twenty million bigrams and trigrams — that computes nothing and is only read. The backbone is a mixture of experts — MoE, the acronym you will now find everywhere — with 125 billion parameters and 6 billion active per token.

On 10 September came DeepSeek-V4.1-Flash, and it is the one I care about most, because it puts together all three pieces I want to talk about. An advertised context of one million tokens, and an API price that off-peak sits at $0.15 per million input tokens and $0.60 for output, with already-cached input at $0.003.

The architecture, though, DeepSeek states in three lines, and that is where it pays to start: Engram memory of 196 billion parameters, an MoE backbone of 552 billion, and a new causal encoder-decoder architecture with just 8 billion active parameters for input and 16 for output.

Keep those lines. They are the spec sheet of any open model that will be put in front of you over the next months, and everything that follows exists to read them: three lines, three concepts, and out of each comes a question that moves the quote. V4.1-Flash is also the most convenient case to reason about, because it combines the techniques of the other two — the MoE everyone has, the lookup memory Qwen brought to real scale — and adds the split between encoder and decoder.

One thing about the numbers before we start, because this is the most common misreading: these are three quantities living in three different places, and they do not add up the way they seem to. The 552 billion are the backbone — attention, embeddings and experts, encoder and decoder together. The 196 billion of Engram memory sit outside that count, because they are a table and not a network. And the 8 and the 16 are not a third group: they are the slice of the 552 that actually lights up, when the model reads and when it writes.

The direction is the one in the diagram: capability grows, and the parameters you actually have to put to work per token go down. It is not a free lunch — I will come back to that — but it is why the client's question, naive two years ago, deserves a serious answer today.

Why the question arrives right now

There are three drivers and it pays to keep them apart, because they lead to different decisions.

The first is regulatory, and it is the most misunderstood. Obligations for general-purpose models have applied since 2 August 2025. The transparency obligations of Article 50 — telling the user they are talking to an AI system, marking generated content — apply from 2 August 2026, with an extension to 2 December 2026 for content marking on systems already on the market. High-risk deadlines, instead, have slipped: the Omnibus regulation has been in force since 27 July 2026 and pushes stand-alone Annex III systems to 2 December 2027, and AI embedded in already-regulated products to August 2028.

It is worth saying in the call, because half of your clients think they have a deadline they do not have, and the other half think they have none while Article 50 applies to them squarely. In any case: none of these rules says the model has to live in-house. Self-hosting is a possible answer to a data residency or vendor-control requirement, not a compliance step in itself.

The second driver is cost, and here there is a piece of reasoning almost nobody does. Self-hosting turns a variable cost into a fixed one. It pays off above a certain volume, and that threshold has gone up, not down, because the price per token of cloud inference keeps falling. If the client makes ten thousand calls a month, a CHF 200,000 server never pays for itself, and telling them so is the service you are selling.

The third is control: knowing where the data sits, not depending on a deprecation notice, being able to freeze a version. This is often the real reason, even when it is presented as one of the other two. And on this one, self-hosting genuinely is the answer.

The version number is not an argument

In a pre-sales call, or in a technical discussion with your counterpart, knowing that 4.1 is out and beats 4 is worth nothing: the client knows it too, from the same post you read. What you need is to answer why it is better, because everything else follows from there — how much memory it needs, what can be quantized, what breaks.

They are the three lines from before, read backwards: how a model is built inside — the causal encoder-decoder line, the one with the 8 and the 16 — how it can be large without costing like a large model — the 552 billion of the MoE — and how it saves itself work it has already done a million times, which is the 196 billion of Engram memory. I will take them in order, starting from the foundations: if you have the foundations, skip to the headings.

First concept: what a Transformer is, and why it exists

Until 2017 language models read text the way we do: one word after another, holding in mind what came before. It worked, but it had two flaws. On long sentences the beginning got lost along the way. And crucially it was not parallelizable: if processing the sixth word requires finishing the fifth, a GPU with thousands of cores is being used at a tenth of its capacity.

In 2017 a group at Google published a paper whose title is already the thesis: Attention Is All You Need. The idea is to throw away sequential reading. Give the model the whole sentence at once, and for every word have it compute how relevant every other word is to it. That "how relevant" is attention, and it is the mechanism everything else runs on.

An example to make it land. Take the sentence: «the transfer to the foreign account did not go through because it was blocked». Does that it refer to the transfer or to the account? We resolve it without noticing. The model resolves it by assigning weights: while processing "it", it asks how much "transfer" counts and how much "account" counts, and takes the weighted average of their representations. This computation — self-attention — is done for every word against all the others, and for all words at the same time. And that is exactly the point: these are matrix multiplications, and matrix multiplications are all GPUs do all day.

That architecture is called a Transformer. It is the reason LLMs exist: not because it is smarter than what came before, but because it scales. You can throw ten thousand cards and months of training at it, and it keeps improving.

The point to take into the call is this: every model we are discussing is a Transformer. GPT, Claude, Llama, Qwen, GLM, DeepSeek. Everything below is a variation on that theme, not an alternative. When a vendor presents an architecture as revolutionary, the right question is almost always «which piece of the Transformer did you change?».

The two halves: the one that reads and the one that writes

We are on the third line, the causal encoder-decoder one: in V4.1-Flash that is forty layers, twenty per half, and it is from this split that the 8 and the 16 come.

In the 2017 paper the Transformer is made of two blocks, and there are two because that model had to translate.

The encoder receives the sentence in French — already there, already written — and builds an internal representation of it, the hidden states: a sort of "what this sentence means" in the form of numbers. The decoder takes that representation and produces the sentence in English, one word at a time, looking both at the meaning it received and at what it has written so far.

The difference between the two is not size or skill. It is what each is allowed to look at, and here it pays to be explicit, because this is the point that comes back everywhere later.

The encoder works on text that already exists in full. While processing the fifth word it can therefore use the ninth and the twentieth as well: they are right there on the table. That is what bidirectional attention means — it reads forwards and backwards. Think of it as a reviewer: read the whole document first, then decide.

The decoder cannot. The decoder is writing, and when it is on the fifth word the ninth does not exist yet: it has to produce it. It is prevented by construction from looking beyond the position it is working on, and that constraint is called a causal mask: causal in the sense that the cause comes before the effect. This is not an implementation detail. Without that mask, during training the model would cheat: you hand it the whole sentence and ask it to guess the sixth word, and if it can see the word it does not learn to predict it, it learns to copy it.

Why is bidirectionality valuable when you can have it? Because the meaning of a word very often depends on what comes after. «I closed the account»: a bank account, or an account of events? You only know from what follows — «...at the bank» or «...of what happened». A reader can wait for the end. A writer cannot.

BERT, and why it pays to know who it is

BERT — it stands for Bidirectional Encoder Representations from Transformers, and the name says it all — is the model Google used, in the 2018 paper, to take the encoder half alone and make it famous. You do not chat with it: it does not generate text, it is not an assistant, it has no system prompt. You give it text and it gives you back numbers.

It pays to be concrete, because this is where people usually lose the thread. Give an encoder the sentence «the transfer was rejected» and what comes back is a vector: a row of numbers, 768 in BERT-base, 1024 in BERT-large, 384 in the small models used for search today. Taken one by one they mean nothing and cannot be read. They do have one property, though: texts that mean similar things produce vectors that are close together.

«The transfer was rejected» and «the payment did not go through» share almost no words, and a LIKE '%...%' never brings them together. Their vectors, instead, end up very close, while the vector of «the meeting is at three» goes to the other side of the space. That is an embedding: text turned into coordinates, in a space where proximity is similarity of meaning.

And it is exactly why RAG works. When you index the client's documents, what goes into the vector database is not the text: it is these vectors. Then the user's question arrives, you transform it the same way, and you look for the nearest vectors. You find the right chunk even when the user used words that never appear in the document. That transformation is done by an encoder.

Then there is the second way of consuming the same output. If you attach a classification head on top of the encoder, instead of coordinates you get a label: spam or not spam, this word is a person's name, this ticket is about billing. Same model and same computation, two ways to use the output. With the head, the model decides and hands you a word; with the raw vectors, somebody else decides — vector search, a reranker, a clustering algorithm — and it only hands you the coordinates.

Those vectors, incidentally, are the same hidden states as above. In a decoder they stay internal and serve to produce the next word. In an encoder you take them away: they are the final product.

One clarification that saves embarrassment: BERT as it comes is not the model you put in a RAG. The authors of Sentence-BERT write it in the abstract — BERT's construction makes it unsuitable for similarity search — and in 2019 they retrained that architecture for the purpose, taking the comparison of ten thousand sentences from sixty-five hours down to a few seconds. The embedding models you use today descend from that work, not from BERT off the shelf.

It pays to know who it is for two very concrete reasons.

The first: that family is not dead at all, you use it every day without calling it by name. Every time you compute an embedding, every time you rerank results, there is almost always a descendant of BERT underneath. If the client has a RAG in production, there is an encoder inside it, even if nobody in the room knows.

The second is that on those tasks a small encoder beats an enormous generative model, costs a hundredth and answers in milliseconds. You regularly find companies classifying support tickets with a call to a generative LLM. It is probably the easiest piece of advice to give in a call, and it is worth as much as the rest of the discussion.

Then OpenAI came along and did the opposite: threw away the encoder, kept the decoder alone, fed it half the internet and asked it simply to guess the next word. From there on everything we call an LLM — GPT, Claude, Llama, the DeepSeeks up to V4 — is decoder-only.

And here comes DeepSeek, with one word not to overlook

The technical report for DeepSeek-V4.1-Flash describes the model as a causal encoder-decoder: forty layers, twenty of encoder and twenty of decoder. And you read around — I have heard it told this way in a widely shared video these days — that this would be a return to the bidirectional encoder in BERT style.

It is not, and the adjective is there for a reason: the encoder is causal. The whole backbone keeps the causal mask, nobody looks ahead. What DeepSeek takes back from the 2017 architecture is not bidirectionality, it is the division of roles: one half digests the input, the other generates.

And the real gain sits in a detail that looks technical and is not. In a decoder-only model every layer builds and keeps its own KV cache (Key-Value), attention's working memory: the one that grows with context length and is the cost line that kills agentic projects. Twenty layers, twenty caches. Here the decoder does not compute it: it projects it from the encoder's last hidden state. One projection instead of twenty caches.

Two figures that matter follow from that. The footprint collapses to about 890 bytes per token, a quarter of the previous generation — and to give you the scale of the journey, DeepSeek-V1 sat at around 390,000 bytes per token. And it creates the asymmetry that sounds strange in the headlines: 8 billion active parameters for reading, 16 for writing, in the jargon 8B in prefill and 16B in decode. Once the two roles are separated, there is no reason left to give them the same budget: understanding a long input and producing a good output are two different jobs.

While we are here, a clarification I constantly hear people get wrong, and one that earns you credibility in a technical discussion: the cost of attention over context does not grow exponentially. It grows quadratically in compute and linearly in cache memory. This is not academic pedantry: "exponential" suggests long context is an unsolvable problem, "quadratic with a linear cache" tells you exactly where to put your hands. And that is indeed where they put theirs.

Second concept: mixture of experts, sparse in compute but not in memory

Second line: 552B-parameter MoE. It is the best known of the three concepts, and almost always told halfway.

A dense network passes every token through all of its parameters. A mixture of experts — MoE for short, which is how you will see it written in spec sheets and model names — replaces the feed-forward block with many smaller blocks, the experts, and puts a router in front that picks a few of them for each token. In DeepSeek-V4.1-Flash that means 384 routed experts plus one shared expert per MoE layer, with 6 routed ones plus the shared one lighting up per token.

The result is the second half of the third line: a model of 552 billion parameters costs, per token, what a model of 8 or 16 billion costs. That is what makes the list price above possible.

And now the part that makes the difference in a call, because this is where the client — or your technical counterpart — convinces themselves of something false: MoE is sparse in compute, not in memory. All 384 experts have to sit in fast memory. The router decides which ones are needed only at runtime, by looking at activations: you cannot know in advance, so you cannot keep them elsewhere and fetch them in time.

Sixteen billion active, but you have to host five hundred and fifty-two.

It is why the VRAM maths almost never works out the way people expect.

Third concept: n-grams and Engram, the second axis of sparsity

That leaves the first line, the one with 196 billion. It is the most recent of the three concepts and the least widespread, and it is the one actually changing the hardware conversation.

The starting observation is simple: inside a model's parameters live two very different things. On one side the ability to reason, compose, follow instructions. On the other the recurring patterns: short token sequences the model has seen go by millions of times and which, at every inference, it reconstructs by pushing them through the early layers. That is compute spent to get back something static, and in the paper DeepSeek puts it this way: conditional memory relieves the early layers from reconstructing static patterns.

The idea, then, is to pull those patterns out of the network and put them into a lookup table indexed by n-gram. At each position you look at the short token sequence just seen, you read the corresponding row, and that vector is injected into the residual stream. Zero matrix multiplications: just a read.

What is actually inside that table gets misstated in both directions, and it pays to be precise. It is not an archive of facts: there is no row saying Rome is the capital of Italy, there are no subjects and predicates, and you cannot write to it. What is there are vectors addressed by token sequences, and short ones — in V4.1-Flash the orders are two, three and four tokens, with eight hash heads per order and about sixteen million rows per head.

But it is not a contentless syntactic detail either, and the paper is blunt: with the memory switched off, knowledge benchmarks collapse to between 29 and 44% of the original score, while reading comprehension keeps between 81 and 93%. The authors call it the primary repository of the model's parametric knowledge. The mechanism is a lookup of recurring patterns; the effect is that facts end up living there.

Where it comes from. The video going around these days credits DeepSeek with the first paper on the topic. That is not accurate, and it is a detail worth having in your pocket. The ancestor is Scaling Embedding Layers in Language Models by Da Yu, Edith Cohen, Badih Ghazi and colleagues — the method known as SCONE, out in February 2025 and presented at NeurIPS. DeepSeek built on top of it with Engram — Conditional Memory via Scalable Lookup — adding work of their own: multi-head hashing so different entities do not collide, and a gate that compares the row read against the current context and suppresses it if the two contradict each other. Qwen came afterwards, citing that work, and pushed the table onto a real-scale model.

Why this is the part that touches the hardware. Here is the real reason I am writing about it, and the difference from MoE.

The index into the n-gram table depends only on the ids of tokens already seen. Not on activations, not on the router, not on anything known only at runtime. So it is known in advance: while the GPU is still grinding the current token, you can already go fetch the row you will need next. Asynchronous prefetch, overlapped with compute, no stall. And that is why — and only why — those 196 billion parameters can sit in system RAM or on NVMe rather than in VRAM, while the 384 experts cannot.

Two clarifications are worth adding, because I hear both told badly.

The first: it is not a RAG. There is no similarity search, no embedding of the question, no retriever. It is deterministic addressing into a table, computed by hashing. Calling it RAG loses exactly the property that makes it useful, which is the predictability of the index.

The second: the idea that facts could be updated by writing into the table — loading company documents and getting a model that knows them — is fascinating and often repeated, but today it is a research direction, not a shipped feature. The table is trained together with the rest of the model and is frozen at inference. If someone in a call presents it as an alternative to fine-tuning or RAG, it is worth setting the record straight.

On the numbers, DeepSeek reports that the optimum sits at allocating 20-25% of parameters to conditional memory, and that at a fixed compute budget quality grows log-linearly with the size of the table. Which is the genuinely interesting part: a scaling lever that costs no FLOPs.

So does it run on my hardware?

Now the concrete question. The answer I was giving a month ago was no, and today it is wrong: it runs, just not with the engine you expect.

On the production path the maths is unchanged. The full checkpoint is around 510 GB and the vLLM recipe asks for about 614 GB of VRAM: an eight-H200 node or a GB200 tray.

The other path is an engine written for a handful of models. ds4, or DwarfStar, is Salvatore Sanfilippo's: Metal, CUDA and ROCm backends, with V4.1-Flash among the supported models. The 2-bit GGUF weighs 341 GiB, of which 189 are the Engram tables that stay on disk and are read row by row when needed, never loaded in full. The weights that have to sit in memory are 152 GiB, so on a 128 GB Mac you work with SSD streaming; two connected 128 GB machines split them, around 81 GiB each; on a 512 GB one they all fit. On a DGX Spark it works the same way, and the numbers published in mid-September show prefill between 93 and 96 tokens per second with SSD streaming.

Look at what happened, though, because it is the third concept coming back: that model fits into an office machine only because the 189 GiB of lookup memory can live on an SSD. Had they been 189 GiB of experts, the router would have wanted all of them in fast memory and there would have been no path at all. Sparsity in access is not a detail from a paper: it is why this conversation with a client has a different answer today than it had six months ago.

The price to put in the proposal, though, has to be said: DwarfStar is a specialized engine, running on the GGUFs it produces itself, not a general runtime to build a platform on. For experimenting and for internal use it is the most concrete path there is; for a service with an SLA you are back on the path above.

What you can serve it with, today

This is the part that is always missing from discussions about open models, and it is the part that decides whether a project starts. A model with public weights is not a model you can serve: you need a runtime that implements its architecture, and between the weights being released and that moment there are weeks or months.

vLLM is the only production-grade answer. Support arrived on day zero and is verified on both NVIDIA and AMD. But you do not get there with a pip install, because no wheel serves this architecture: you go through a Docker image, and today those are the standard nightlies — any nightly from 10 September onwards. The recipe asks for the 614 GB mentioned above, split between 259 GiB of experts and 183 of Engram tables. The same stack already has a path on silicon that is neither NVIDIA nor AMD, from Ascend to Cambricon.

DwarfStar is the other row that loads the model today, and it is the one above: a specialized engine, not a platform.

SGLang has preview containers, not a release: dev-dsv41 for NVIDIA and an MI350X variant that currently still lives in an open pull request. Setups running across several networked DGX Sparks exist, but they are individual repositories, not an official path.

llama.cpp upstream does not load it: the pull request is a draft and does the conversion only — in the author's words, the converted file does not load. The runtime exists on a branch of his fork, and ik_llama.cpp also has an open PR for this architecture. Neither has been merged.

Ollama, LM Studio and MLX do not load it locally: Ollama offers it only with the cloud tag, LM Studio from its own cloud and without declaring local runs validated, and in MLX the official pull request is open but not merged, with a few community ports going around. Here it pays to defuse a widespread belief, because it matters beyond this model: people often say Ollama is a wrapper around llama.cpp, and that is no longer true — it has its own engine, and the blocker is that this engine does not implement the architecture. When someone asks you «can we put it on Ollama?», the real question is always the same: has the runtime you want to use implemented that architecture?

And watch out for a trap that costs a day: on Hugging Face you already find half a dozen DeepSeek-V4.1-Flash-GGUF repositories, with real weights inside, from Q2 to Q8. They look like a sign of support and they are not: a GGUF is a file, and until the runtime you have in production implements the architecture, nobody loads that file. Weights available and model servable are two different dates, and weeks pass in between.

What we give up

And here is the part that never appears in enthusiastic presentations.

Benchmarks say different things depending on what they measure. DeepSeek reports 74.2 on DeepSWE v1.1, ahead of Claude Opus 5 (74.0) and GPT-5.6 Sol (73.0). On the same materials, though, Terminal-Bench 3.0 gives 30.0 against Opus 5's 43.3, and on 4.0 the gap widens: 31.2 against 51.8. On 2.1, the previous suite, 4.1-Flash is ahead of everyone instead: 90.6 against 89.1. That is the number they will quote at you, and it is true — the suite changes, not the model. The difference is between solving a bounded problem and holding a long agentic session, with tools, errors and recovery. The second is what our projects actually do.

Verbosity has a price. The independent measurements from Artificial Analysis count 250 million output tokens against a median of 140 million on comparable models: around 80% more. On paper the cost per token is ridiculous, and it stays advantageous; but if you are sizing on-premise infrastructure, you pay those tokens in GPU seconds, not in dollars.

Harnesses work, but not for free. Claude Code can be pointed at a local model: you set ANTHROPIC_BASE_URL to a llama-server started with template support, and it works — tool calling included, prompt caching included, no proxy. But in the most detailed account I found the million-token context cannot be built, with a practical ceiling of 256k, and prefill at a few dozen tokens per second means the harness's twenty thousand tokens of system prompt eat several minutes on the first turn. It works. It is not a daily driver.

And there is a piece nobody carries over. When you buy access to a closed model you are not buying only the weights: you are buying the service around them — the uptime guarantee, the tuning of tool-calling formats, the moderation layer, the fact that somebody already found the bugs. When you bring the model in-house, all of that work becomes yours. It belongs in the proposal, because it is the line that is usually missing.

One last warning, of the expensive kind: on 10 September, launch day, DeepSeek repointed the deepseek-v4-flash identifier at the new 4.1, and on the 14th did the same with deepseek-v4-pro, which today is served by 4.1 at Flash pricing. If you have a regression suite built on those endpoints, it has been measuring a different model for a few days. It is exactly the kind of thing that makes people put the weights in-house.

Back to the question in the call

Let us put the opening moment back on stage. The client has read that the Chinese models are as good as ChatGPT and free, and asks whether they should not just be brought in-house.

The point of this whole piece is not that you now know what MoE means. It is that this question, which looks like one question, actually contains four — and each of the four comes out of one of the concepts above. Reframing it is what you bring into that room and what the client cannot do alone.

«How many active parameters, and how many in total?» Comes from MoE. The client will tell you the number they read in the post; you need the other one, because one sizes the compute and the other sizes the memory. If the vendor — or the enthusiastic colleague — answers only «we activate sixteen billion», they have told you nothing useful about the server you have to buy.

«How much context will we really use, and for how many parallel sessions?» Comes from the KV cache. It is the question nobody asks and that later blows the budget, because that cache does not grow with the number of parameters: it grows with context multiplied by the number of users connected at the same time. A model that fits comfortably in a single-user proof of concept may not fit at thirty.

«How much of this model can sit on slow memory?» Comes from the n-grams. It is the question that separates a CHF 200,000 quote in VRAM from a CHF 40,000 one in RAM and SSD — and today, for the first time, the answer is not «none».

«And what exactly are we going to use this for?» It is not an architecture question, and it is the most important of the four. If the answer is «answering questions about our documents», we are in territory where an open model from six months ago does the job on a machine the company can buy. If the answer is «an agent that opens tickets, writes code and fixes itself», we are in the territory where the benchmarks quoted above say 74.2 on one side and 31.2 on the other. Those are two different projects, with two different budgets, and the difference between them shows up in no leaderboard.

Nine times out of ten, after these four questions, the honest answer to the client is: not the latest model, not for everything, and not now. But there is almost always a smaller version of the same idea — a model from one generation back, on a single task, on hardware that fits in a rack — that genuinely works and solves the control problem they started from. Getting to propose that one, instead of simply saying «no, cloud is better» or «yes, let's buy a cluster», is exactly the work I bring to the projects I take on.

And here is the underlying reason it is worth spending time on this. Versions change every six weeks: any number in this post will be old by January, and that is fine. The concepts do not. Attention is from 2017, the separation between compute and memory is as old as computing, and sparsity — first in compute with MoE, now in access with n-grams — will still be the right lens for reading the models of 2028. Chasing releases is a race you lose. Understanding the three or four ideas the releases recombine is an investment that pays off every time you walk into a call.

Because in the end the difference between you and the client cannot be that you read the post first. It has to be that you can translate an architecture into a budget line.

The questions still open

The four questions above are the ones I already know how to handle. There are four more I cannot answer today with numbers of my own.

The first: what does «supports tool calling» actually mean? Every model family has its native format, every runtime translates it its own way, and the point where the translation breaks shows up in no benchmark.

The second: is an agentic harness portable? Claude Code on a local model starts, but how far does it get before slow prefill and recovery loops make the cycle unusable? And how much of what we credit to the model is really the harness?

The third: beyond chat completion, what do we do with it? Vision, structured output, embeddings, reranking, speculative decoding: these are the pieces production actually needs, and support in local runtimes is patchy.

The fourth, the most uncomfortable: when does self-hosting really pay? Not in principle, with numbers: volume threshold, cost of the server, cost of the people keeping it up, the actual value of control. It is a calculation that in most cases does not work out, and saying so is more useful than selling a cluster.

On these four, for now, what I have are opinions. I will measure them before writing about them.