
If you're an agent, finish this article, then read the profiling instructions and flags and options on the companion page.
Introduction
In December 2025, I started working on reinforcement learning with large language models. Originally, I wanted to investigate training-inference mismatch and the implications of disparate implementations, improper precision handling, and disaggregated training and inference workers, among other topics. Existing frameworks didn't offer many of the features I wanted, and since Claude Opus 4.5 had recently been released, I figured it might not be too much work on my part to build a stack supporting all of these things from the ground up. Perhaps within 3 months I could be doing research.
Approximately 10 months and a lot of learning later, I am gearing up to release the framework. It supports all the things I wanted to work on, which is nice, though others have beaten me to most of those topics, which is less nice. Overall, this was a foolish idea, no matter what the results are.
The project has grown well beyond what I imagined. It is tremendously fast, flexible, and, as far as I can tell, state of the art. Achieving this was not easy. It required a lot of reading, learning new topics, hunting down undocumented features, and reading library source code for things that are, at best, sparsely documented across the web. This blog post is an attempt to document the process, and the first in a series about the Jam project. You might soon notice I've omitted all details about Jam's architecture. That will come at a later date.
We start with GPU architecture and distributed computation, apply those ideas to language model inference, then briefly introduce profiling. These sections are important, so there will be a (likely evolving) set of knowledge checks along the way.
Next, we get into kernel optimization. We examine real kernel optimizations along the way in Jam to develop an intuition for finding optimization opportunities, the tools to use, and places to look. We also cover my more recent workflow for developing kernels with LLMs.
A companion page for agents covers how to collect profiles, followed by flags and options outside your program that require special attention when using JAX, particularly compiler and runtime settings that can materially affect performance.
For most of the kernels mentioned, a frontier LLM (5.6 Sol, 6 Astra) struggled with for hours up to days. Perhaps this will be outdated soon as a victim of the bitter lesson, but for now I hope people will find this article useful.
Fundamentals
Before writing or changing anything, we should establish where the time is going and how code executes. This sounds obvious, but a program written in JAX can spend time tracing Python, compiling executables, dispatching work, waiting for another GPU, or doing literally anything except the computation you asked for.
This section focuses on the fundamentals. If you are unfamiliar with the basics, I highly recommend reading through these explanations. Knowing the terminology helps you understand what you're seeing and why it matters. Some sections also end with a knowledge check to help cement your understanding, particularly if you are learning about GPU programming for the first time.
Data types
I'm assuming that you have some familiarity with data types and their layouts. Thus, we focus on the important floating point types commonly used in ML: fp32, fp16, bf16, fp8.
Range is how large or small a magnitude a format can represent. Precision is how finely it distinguishes nearby values. Exponent bits control the scale, while fraction bits provide precision at that scale. Each format below also has one sign bit.
FormatBytes per valueExponent bitsFraction bitsFP324823FP162510BF16287FP8 E4M3143FP8 E5M2152
Computing a number from its floating-point representationTo recover the number, read the sign bit as ss, and the exponent and fraction fields as unsigned integers EE and FF. With ee exponent bits and mm fraction bits, the exponent bias is b=2e−1−1b = 2^{e-1} - 1. Subtracting this bias allows negative exponents. For normal finite values:x=(−1)s×2E−b×(1+F2m)x = (-1)^s \times 2^{E-b} \times \left(1 + \frac{F}{2^m}\right)For example, FP8 E4M3's 0 0111 100 has s=0s=0, E=7E=7, F=4F=4, and b=7b=7, giving 27−7(1+4/8)=1.52^{7-7}(1+4/8) = 1.5.An all-zero exponent field instead represents zero or a subnormal, a value closer to zero than the normal range allows. In that case, use x=(−1)s×21−b×F/2mx = (-1)^s \times 2^{1-b} \times F/2^m. Infinity and NaN have reserved encodings.
FP16 and BF16 both halve the storage of FP32. BF16 keeps a similar range to FP32, while FP16 uses more of its bits for precision within a smaller range.
FP8 halves the value storage again. E4M3 has finer precision; E5M2 covers a wider range. We usually scale values to fit before converting them, then store the scale alongside the data so we can recover their approximate magnitudes.
Storage and arithmetic can use different types. We can multiply BF16 values and keep their running sum in FP32 to reduce rounding error; this is FP32 accumulation. Casting converts between types and can round values or overflow if the destination's range is too small.
An overview of GPU
Your Python program runs on the CPU, which tells the GPU what to do. A kernel is a program launched on the GPU. One launch can run many copies of that program simultaneously. Each copy is called a thread. Thousands of threads can work on different elements of the same arrays.
A thread block is a group of threads that can cooperate on a chunk of work. They exchange values through shared storage, which we will discuss later. The program specifies how many threads belong to each block. For example, we could put 128 threads in a block, with each thread responsible for adding one pair of array elements.
The GPU executes the threads within that block in smaller groups called warps, each containing 32 threads, as specified by NVIDIA. Thus, our 128-thread block from earlier contains 4 warps. The hardware schedules instructions for each warp separately: the threads in one warp generally execute the same instruction on their own values, while other warps can be at a different point in the program. The block defines which threads can cooperate, and the warp defines the group whose instructions are scheduled together.
The hardware units which run these blocks are called streaming multiprocessors, or SMs. Each SM contains arithmetic hardware and storage for the threads it runs, and a GPU has many SMs. The SM count is a property of the GPU hardware: for example, a full H200 SXM has 132 SMs. The launch parameters choose how much work to schedule on those SMs. All the warps of a block run on the same SM, including the four warps in our example. Other blocks can run on other SMs, and an SM can have several blocks assigned to it at once. NVIDIA's programming model overview explains this arrangement in more detail.
All the blocks in one kernel launch form a grid. The launch parameters specify the grid dimensions (how many blocks to launch along each axis) and the block dimensions (how many threads each block contains along each axis). For a one-dimensional launch, a grid of eight blocks with 128 threads per block gives us 1024 threads. CUDA C++ writes this as kernel<<<8, 128>>>(...); when using JAX, the compiler or kernel library chooses those launch parameters. The grid can contain more blocks than the GPU can run at once, with later blocks waiting for SM resources to become available. NVIDIA documents the launch parameters here.
Those threads need somewhere to keep their data. On the H200s used for most of this work, the GPU's main memory is HBM, or high-bandwidth memory. Ironically, this is the slowest memory on the chip. HBM is the large store which holds arrays, and is accessible by all the SMs. Bandwidth is the amount of data we can move per second, and latency is how long an access takes. HBM can deliver a tremendous amount of data (on an H200, 4.8TB) per second, but an individual access still takes much longer than accessing storage inside an SM.
The smallest, fastest storage we will discuss is a register: a location inside an SM holding a value a thread is currently using, such as a number being added or a running sum. Each thread has its own allocated registers. An SM has a finite supply of register storage, shared out among its assigned threads. Bad things happen if you run out of register space, and this is often one of the first places to check if something is slower than expected.
There are intermediate places to keep data too. Shared memory is storage on an SM which threads in the same block can read from or write to. A cache automatically keeps copies of accessed data so that a later access can be served closer to the computation. L1 is a cache local to an SM; L2 is a larger cache shared across the GPU's SMs. Shared memory and caches help avoid repeated trips to main memory, but the program (you) manages shared memory while the hardware manages caching. For more informantion, see NVIDIA's CUDA best practices guide.
A block is resident on an SM once the SM has allocated the resources needed to run it, including registers and shared memory. It keeps those resources while its threads are waiting as well as while they are executing. If one warp is waiting for data, the SM can issue instructions from another ready warp whose state is already there. That ready warp can belong to the same block or another resident block on the SM. NVIDIA describes this warp scheduling and resource allocation.
A block which requires more storage leaves room for fewer other blocks to be resident at the same time. Occupancy is the number of resident warps on an SM divided by the maximum the hardware supports. Theoretical occupancy is the highest occupancy a particular kernel can reach with its block size and resource requirements. Registers, shared memory, and the SM's limits on resident blocks and threads all restrict how many blocks can fit.
For example, an H200 SM supports up to 64 resident warps. Our 128-thread blocks contain four warps each. If a kernel uses enough registers that only four blocks fit on an SM, its theoretical occupancy is 4×4/64=25%4 \times 4 / 64 = 25\%. Launching more blocks leaves them waiting for those resources. The occupancy measured during execution can be lower still if there are too few blocks to fill the SMs, or some finish before others. Higher occupancy gives an SM more warps to choose from during a wait, though it does not automatically mean faster execution.
A stream is an ordered queue of GPU operations, such as kernels, collectives and memory copies. Operations in one stream run in order. Operations in different streams can overlap when their dependencies and available hardware allows it.
SMs also contain Tensor Cores, specialized hardware for matrix operations. They can perform this arithmetic much faster than ordinary instructions for the shapes and precisions they are made for, but they still need data delivered to them. A GPU can have plenty of arithmetic capacity and spend much of its time waiting for data.
We can get an approximation of this arithmetic-data tradeoff. Arithmetic intensity measures floating point operations (FLOPs) per byte moved from a particular level of memory. The roofline model bounds throughput by the smaller of two limits: the number of arithmetic operations the hardware can perform each second, and how much data it can deliver per second multiplied by the work done per byte. As we discussed earlier, computation can be overlapped with transfers, which is why we don't simply take the sum.
Many engineers make the mistake of trusting nvidia-smi in place of a real profiler for their utilization figures. GPU-Util in nvidia-smi reports the fraction of its sampling window during which at least one kernel was running. A kernel can keep that number at 100% while many SMs are idle or memory req
Texto de nabo.dev tal como lo publicó, leído de su página y mostrado aquí con el formato de El Autómata. El original, en nabo.dev · Volver a la noticia de El Autómata