Make this forward pass as fast as possible
Flapping Airplanes just raised a billion dollars, in part through demonstrating a megakernel framework that could do efficient forward and backwards passes while (for some reason) playing Doom1. El…
Flapping Airplanes just raised a billion dollars, in part through demonstrating a megakernel framework that could do efficient forward and backwards passes while (for some reason) playing Doom1. El…
Flapping Airplanes just raised a billion dollars, in part through demonstrating a megakernel framework that could do efficient forward and backwards passes while (for some reason) playing Doom1. Elsewhere, Hyperstition have what they call a Hyperkernel, which is generating high-performance kernels for all sorts of models. Turning a model into something faster is exactly what ML compilers have traditionally done, and all this activity begs the question of whether we need ML compilers at all any more, or whether we can let agents optimize directly?
The folks working on TVM have been asking this, too:
The core of their argument is that kernel optimization is a search and iteration process. Tokens and tool calls cost money and time. So, you want to optimize that search, avoid getting stuck at local optima, and offer ways to verify the generated solutions are correct. They build checkers for things like memory access and synchronization and they formalized their IR in Lean to enable cheap verification:
Together’s new Together Compiler (TKCC) builds optimized inference kernels. But, they reach quite a different conclusion around intermediate representations:
They still focus on the needs of verification and cutting down the search space, though. Much of what they built was an empirical measurement framework, with different tiers for executing kernels that normed out variation in forward passes, and offered hot-swapping for fast iteration. They also cut down on search by developing a custom cost model for parallelisms that culls options analytically, before testing in their tiered setup:
So its not so much compilers going away as it is them being decomposed. A lot of traditional compiler techniques become less useful in a world where inspection and investigation is a cheaper, and a lot more automated.
At the Triton conference this year, Neil Dhar from OpenAI2 talked about how they have been using Codex to do large scale code-mods, with the aim of removing heuristics and implicit semantics from the Triton compiler. Those kind of techniques are great for getting improvements across a wide range of code, but they can make the code unpredictable: a small change can have an outsized effect on the result because it triggers different compiler passes.
One example Neil used was removing automatic FMA. Triton had pattern-matched a multiply followed by an addition, and replaced it a single fused multiply-add. But! That could change the numerics of the kernel, due FMA dropping intermediate rounding. They removed this pass and manually inserted FMA calls in the source of all the kernels that used it3. This makes the search landscape smoother, and easier to navigate (though possibly larger)
TVM’s TIRx is a bet that starting from fresh will allow them to build a more search-and-verification friendly compiler stack: they deliberately don’t apply heuristic-based optimizations, but do offer many extension points so agents can iterate new capabilities on top. OpenAI’s Gluon and Meta’s TLX are both directionally similar bets: they add low-level extension points while reinforcing the core of an existing compiler. Together are suggesting that providing the right tools is more important than betting on a single toolchain to guide search and verification.
Compilers users to make things faster, but now they need to make the search faster, which isn’t quite the same thing. I would be hesitant on pronouncing a winner though, this space co-evolves with the agents themselves. In one particularly clear example Together had a note about some of the things that absolutely didn’t work in helping their agents: it read pretty much exactly like a list of agentic best-practices, circa six months ago4.
Texto de ianbarber.blog tal como lo publicó, leído de su página y mostrado aquí con el formato de El Autómata. El original, en ianbarber.blog · Volver a la noticia de El Autómata