[AINews] Megakernels are so dead and so back

Part of our Inference Engineering Masterclass pod yesterday involved a spicy discussion about Megakernels: megakernels are dead why are megakernels useful? you spend two months writing a kernel to save time on launch overhead and poor inter-kernel overlap. you had PDL but then people said it wasn't perfect, that you could still get some marginal gains due to straggler CTAs and therefore- wait, sorry, I forgot, Rubin fixes that (kernel two needs 10 CTAs and kernel one has seven finished and three straggling, kernel two launches seven of its CTAs). given a long enough timeline, it all evens out. no serious inference provider is using a 67k loc hand-fused forward pass kernel in production, and the teams doing that are doing so out of pure research. dead. @swyx 's pod and said some things that i... should not have said. \n\na lot has happened since then, i owe you all an apology.\n\ni'm sorry that i was right about every single thing. \n\na) re megakernels are dead\nwhy are megakernels useful? you spend two months","username":"waterloo_intern","name":"ali","profile_image_url":"pbs.substack.com","date":"2026-08-03T23:49:24.000Z","photos":[{"img_url":"pbs.substack.com","link_url":"t.co"},{"img_url":"pbs.substack.com","link_url":"t.co"},{"img_url":"pbs.substack.com","link_url":"t.co"}],"quoted_tweet":{"full_text":"The Inference Engineering Masterclass: 10x faster models, quantization, speculative decoding, Rubin, & self-optimizing AI t.co @philipkiely and @waterloo_intern explain what actually happens after a model is trained, why turning weights into a fast","username":"latentspacepod","name":"Latent.Space","profile_image_url":"pbs.substack.com"},"reply_count":53,"retweet_count":67,"like_count":1276,"impression_count":380220,"expanded_url":null,"video_url":null,"video_preview_media_key":null,"belowTheFold":false}" data-component-name="Twitter2ToDOM"> The full discussion, for those who care to listen through: Ali: A fused kernel can’t save you . Like here with tensor parallelism, half the matrix is on one GPU and the other half is on another, and if I need the entire matrix in order to do like a nonlinear operation in the next step, which is, for instance, like if I’m doing attention, I need the softmax, or I need to do like exponentiation, I need to have the entire row. So I need to know what the partial result was from GPU 2 and what the partial result was from GPU 1 in order to be able to do the softmax in the next stage. So I have to make them communicate with each other, even if I had a fused kernel, because of the nonlinearities within each one . Also with like mega kernels, like honestly, I’m very bearish. It was a good research direction, and it seems like intuitively, theoretically, it’s nice. You have a lot of launch overhead from launching- Just- one kernel- Yeah, just keep fusing it and moving the data. Just fuse everything together. But the kernel complexity itself is very difficult to write a very optimized mega kernel. It’s very difficult to do so. And not to name any companies , but like even the companies that have worked or people that I’ve spoken to who work at companies that do fused mega kernels, they very often don’t end up running those in production because the TensorRT-LLM and modular kernels that launch are faster because you can optimize each individual component, and you can just have them parallelize with each other. One of the tech leads at NVIDIA launched a Twitter post said like, “We’re pulling the curtain on Rubin, and here’s the specs.” And the third tweet showed, like not to get too technical into it, I and I need to read it much more, but the GPU is designed in such a way that it kills mega kernels. So it seems like that entire research field won’t be continued. He was quoting ( friend of the show! ) Kyle Kranen announcing dependency triggers - one part of the pipeline blockage that previously justified kernel fusion: As voiced on the show, there are still physical constraints that are unanswered, but it makes complete sense that Nvidia is updating Rubin design to better fit macabre things that are being done in kernel-land. One of Ben Spector’s megakernel coauthors , Stuart Sul, is now leading the team that released Mixture of Kittens (a reference to Ben’s delightfully named ThunderKittens , and part of Dan Fu’s group ), Cursor’s open source megakernel today: Headline results are compelling - a 41% increase in overall tokens per second. At scale, this translates to billions of dollars worth of savings. AI News for 8/3/2026-8/4/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews' website lets you search all past issues. As a reminder, AINews is now a section of Latent Space . You can opt in/out of email frequencies! AI Twitter Recap Frontier Model Releases: Qwen 3.8-Max, Alpamayo 2 Super, Pokee-Isaac, Maple-Preview, and Shieldstral Qwen’s release cadence continues across modalities : @Alibaba_Qwen launched Qwen3.8-Max as “better and cheaper,” and quickly pushed it into agent ecosystems via Hermes Agent , Nous Research , and ClinePass . On the vision side, @skalskip92 highlighted Qwen3.8-Max’s box-conditioned detection behavior, reporting 60% mAP with a single box and 80% with multiple boxes for hard-to-describe concepts; Qwen’s image stack also moved up, with @arena and @Alibaba_Qwen noting Qwen-Image-3.0-Pro reached #5 in the Text-to-Image Arena. NVIDIA and Mistral both leaned into deployable specialization : @JensenHuang introduced Alpamayo 2 Super for AV reasoning with commercial-use open release terms, while @MistralAI launched Shieldstral , a 3B open-weights safety model designed for on-device moderation/classification. @vllm_project shipped day-0 serving support and highlighted one-forward-pass safety scoring, multimodal input, 12 languages , and 32k context . Long-context and efficient-weight experimentation accelerated : @Pokee_AI released Pokee-Isaac 28B , claiming a 10M-token context , 93.3% RULER at 10M , and single-GPU deployability starting from an RTX 4090; the model is also said to have day-0 support in vLLM and SGLang . Meanwhile @deepgrove_ai introduced Maple-Preview , an open-source 20B-A1B ternary-weight reasoning model said to run at 200+ tok/s on a Mac Mini M4 and outperform others in its weight class. Both releases point to a growing split: not just bigger frontier models, but aggressive exploration of context architecture and low-bit/ternary efficiency. Inference Economics, Routing, and Kernel/Serving Infrastructure Pricing pressure is now changing product design : The permanent Luna repricing from @thsottiaux triggered immediate discussion about always-on helper workloads; @theo described Luna as cheap enough to spin up on nearly every prompt for metadata/status generation. In parallel, several posts stressed just how dominant DeepSeek-V4-Flash is on price: @kimmonismus , @AndrewCurran_ , @ollama , and @EpochAIResearch all reinforced the idea that open(-weight) or quasi-open serving economics are now competitive enough to shape stack choices, especially for high-volume agent workflows. Routing is becoming a first-class systems problem : @tomas_hk launch