Source: Conditional Memory via Scalable Lookup — Cheng et al., DeepSeek-AI & Peking University, arXiv 2601.07372
Technical AI papers can look intimidating from the outside. Lots of architecture names, benchmark tables, and acronyms behaving like they pay rent. But the useful question is usually simple: what bottleneck is this trying to remove?
There’s a specific inefficiency in every current large language model that DeepSeek researchers decided to fix, and the result is one of the more interesting architectural papers of early 2026. The problem: Transformers have to use expensive multi-layer neural computation to “remember” static facts — named entities, common phrases, formulaic patterns — that could be retrieved far more efficiently via a simple lookup table. Engram introduces conditional memory as a complementary architecture primitive: N-gram embedding lookup tables that retrieve static knowledge in O1 time, working alongside the existing Mixture-of-Experts computation. The result of a 27B Engram model versus an equivalent pure-MoE baseline: +3.4 MMLU, +5.0 BBH, +3.0 HumanEval. The reasoning gains are larger than the knowledge gains — by freeing early layers from static reconstruction, you get more effective depth for complex reasoning. The long-context improvement is also striking: Multi-Query NIAH retrieval improves from 84.2 to 97.0. When the model stops wasting attention on local static facts, it has more capacity to track global context. This is a real architectural insight, not an incremental tuning result. The U-shaped scaling law the paper reveals suggests an optimal allocation between neural compute and static memory at each model size — and at large scales, even simple lookup mechanisms dramatically improve performance when treated as a first-class primitive.
My takeaway: architecture matters because it decides what kind of intelligence is affordable, reliable, and usable outside the lab. The benchmark is the beginning of the story, not the ending.