The Model Fits. Why Is It Still Slow?

Memory capacity is usually the first obstacle when running a large language model locally. If BF16 weights will not fit, try 8-bit, then 4-bit. Once the weights finally load, it is tempting to think the hard part is over.

Yet generation can slow down again when we introduce long documents, a code repository, or several simultaneous tasks. The model has not run out of memory, and the GPU has substantial theoretical compute. Where is the time going?

Guo, George, and Joshi at Notre Dame offer a useful set of results in their October 28, 2026 preprint, GAAC: about 26% less resident KV memory, 1.19–1.27× faster attention kernels at 64K context on DGX Spark, 1.04–1.15× faster complete W4A16 decode steps, and 13%–29% higher aggregate throughput under fixed KV budgets.

Getting the model into memory clears the first hurdle. What often limits speed next is moving data into the compute units at every step. GAAC is interesting because the same memory saving produces very different gains at the kernel, model, and serving levels.

The weights fit. Historical tokens still need memory.

Weights are only part of the resident data used during inference. A Transformer also retains historical keys and values so that generating the next token does not require recomputing previous K and V states. This KV cache grows with context length and the number of concurrent sequences.

For a typical full-attention architecture, the theoretical storage for one sequence can be approximated as:

KV bytes = 2 × layers × KV heads × head dim
           × context tokens × bytes per element

The initial 2 accounts for K and V. Take a hypothetical full-attention architecture: 32 layers, 8 KV heads, a head dimension of 128, and BF16 storage. Its KV takes 128 KiB per token, or 8 GiB at 65,536 tokens. Four equally long sequences without shared prefixes need 32 GiB for KV alone. Allocator overhead and temporary workspaces come on top.

Weight quantization does not automatically shrink that cache. An 8B model has roughly 16 GB of raw BF16 weights or 4 GB at 4-bit, while its KV may remain BF16. Costs previously overshadowed by the weights become more prominent. GQA, MQA, sliding-window attention, and hybrid attention change that growth, so similarly sized models need not have similar KV requirements.

Caching carries another bill. Each generated token in standard full attention accesses historical keys and combines their values. Previous computation is reused, but reading its results has to be paid for again at every step.

At that point, a high TFLOPS figure can sit idle. Prefill organizes many input tokens into large matrix operations. Small-batch, single-request decode repeatedly reads weights and historical KV. The arithmetic units wait for data while memory bandwidth is already busy.

Suppose a path reads 8 GiB from device memory each step at an available decimal bandwidth of 1 TB/s. The ideal transfer floor is already about 8.6 ms. Cache hits, reuse, and bandwidth utilization change the bill, but reading still costs time. Long-context generation pays it repeatedly.

Where does the compressed cache get unpacked?

FlashAttention already demonstrated the value of reorganizing computation and memory access. GAAC turns to the KV representation itself: if historical states are read repeatedly, can we move fewer bytes without changing the values stored?

BF16 consists of one sign bit, eight exponent bits, and seven mantissa bits. GAAC preserves the sign and mantissa and encodes only the exponent. Model-generated exponents are not uniformly distributed; frequently occurring exponents in many head-channels occupy short intervals. Calibration chooses high-coverage windows of length 8 and 16 for each head-channel of K and V in every layer. Encoding then stores offsets from those window bases.

Eight elements share a 2-bit mode tag. A group uses 3-bit offsets if every element fits the narrow window, then tries 4-bit offsets, and falls back to raw 8-bit exponents if necessary. Outliers take the fallback path. Calibration mismatch costs compression efficiency while preserving the original values.

In the common 3-bit mode, each element takes 8 + 3 + 2/8 = 11.25 bits. About 88%–90% of groups use it across five tested models, yielding complete encoded-size ratios of 1.405–1.407×. At 1.406, encoded storage falls by 1 - 1/1.406 ≈ 28.9%. Fixed slots, block tables, and fallback consume some of that saving, leaving about 26% less resident KV memory. Weights and other runtime allocations keep their own costs.

Making KV smaller is not enough to make inference faster. A straightforward implementation reads the compressed data, decompresses the whole cache into device memory, and hands it to an ordinary attention kernel. That introduces a full-cache write followed by another read. The saved traffic can disappear in the unpacking process.

GAAC fuses reconstruction with attention. The kernel reads compressed tiles and metadata, reconstructs the BF16 values it currently needs, and immediately uses them for attention scores and weighted value sums. It never materializes a complete decompressed cache in external memory. New tokens first enter an incomplete BF16 tail block; a full block is compressed once, rather than repacked with every subsequent token. Blocks that exceed compressed-slot capacity remain in raw BF16.

This scheduling determines whether compression belongs in the inference path. KV is appended, read, and released throughout a request, and historical blocks are consumed repeatedly by later tokens. GAAC preserves original KV bit patterns and reconstructs them where they are used. Integration then compares the floating-point kernel's numerical behavior and task quality against the baseline. That is a more useful target than a prettier compression ratio alone.

More computation, less elapsed time

Table 5 breaks down a decode-attention kernel at 64K context. Times are in microseconds:

GAAC saves 294.3 μs on KV loading and spends an extra 49.2 μs on computation, saving 245.1 μs overall. That is about a 1.22× kernel speedup, or 18% less elapsed time. Extra arithmetic buys a larger reduction in memory traffic.

The decompress-first path takes 2290.1 μs, slower than the uncompressed baseline. The same compressed representation leads to opposite performance outcomes depending on where reconstruction happens and how the result is consumed.

The standalone codec ablation in Table 4 makes the tradeoff sharper. Adding a 2-bit mode improves the ratio from 1.406 to 1.439×, about 2.3%, while decoding latency jumps from 187.1 to 526.7 μs, roughly 2.8×. A small storage saving purchases a much larger reading cost.

On a bandwidth-limited path, giving the GPU more arithmetic can be the route to better performance. The question is how much traffic it replaces and how reconstruction, access granularity, and layout fit together. Optimizing arithmetic and memory traffic in isolation makes it easy to save the wrong thing.

A 1.22× kernel does not make the request 1.22× faster

A decode step also includes linear layers, MLPs, normalization, weight access, and runtime overhead. Suppose attention originally consumes 40% of the total time and becomes 1.22× faster while everything else stays unchanged:

overall speedup = 1 / (0.6 + 0.4 / 1.22)
                ≈ 1.078

Overall processing rate improves by only about 7.8%. Amdahl's law puts a number on the ceiling of a local optimization: the gain depends on its original share of execution time. Map the complete step before deciding how much engineering effort a kernel deserves.

GAAC measures this in its equal-batch experiment. On DGX Spark, attention at batch 1 and 64K context improves by 1.19–1.27×. Complete W4A16 decode steps improve by 1.04–1.15× across the tested model and context configurations. The two hybrid/MoE models use full attention in only about a quarter of their layers and gain around 1.04–1.06× for the complete step. Other layers and weight access still consume much of the time.

For a long document, prefill must process the input first. GAAC's fused path primarily accelerates decode; prefill retains conventional attention and may temporarily expand cached context when necessary. A 64K input followed by a few dozen generated tokens can spend most of its time before the first token. Generating thousands of tokens over that context offers more opportunity for per-step savings to accumulate.

Kernel ratios have their own shape. Qwen3-8B attention gains are 1.31, 1.27, and 1.25× at 32K, 64K, and 128K context. Complete-step gains follow a different curve as the share of KV access changes. Picking the largest multiplier for a headline is easy. Finding where the actual workload sits on those curves is useful.

Does serving one more request count as faster?

A separate experiment fixes the KV-memory budget and increases concurrency to the maximum each system can hold. In Table 2, Qwen3-8B with 64K context and about a 20 GB KV budget fits two BF16 sequences at 19.0 aggregate tok/s. GAAC fits three and produces 22.8 tok/s, a 20% increase.

Dividing the totals by active sequences gives about 9.5 tok/s per sequence in the first setup and 7.6 in the second. Concurrency changes from two to three: the device produces more tokens overall while the arithmetic share per sequence falls. Alongside aggregate throughput, I want measured per-request ITL and completion time to see what the extra capacity buys.

For one person's chat assistant, room for another sequence has limited value. Consider three tasks with separate long contexts: admitting the third instead of queueing it can change group completion time. Comparing such task traces reveals how scheduling and dependencies affect the experience.

One contrast in the paper is especially revealing: the hybrid and MoE models show relatively modest equal-batch gains but 27% and 29% higher throughput under fixed KV budgets. Much of that benefit comes from accommodating another concurrent sequence, not from making each computation step faster by the same percentage.

Online serving also has to account for queueing and tail latency. A larger batch can produce more tokens while leaving some users waiting longer. I would tie goodput directly to the product, for example counting only requests that meet both first-token and ongoing-generation latency targets. Chat needs responsiveness; offline document processing values batch completion. The product determines the target.

What changes if the deployment already uses FP8?

PagedAttention established the importance of cache management. Paged allocation and sharing reduce fragmentation and duplication; its vLLM evaluation reports 2–4× throughput against contemporary baselines on the tested models and workloads. GAAC moves into the representation itself and how attention consumes compressed data. Much of serving performance lives around the matrix operations.

FP8 halves the KV storage width relative to BF16 and introduces numerical approximation. In vLLM's April 2026 report, an H100 Llama-3.1-8B test with 150 requests, concurrency 8, and roughly 20K input/2K output tokens improves aggregate throughput from 450.3 to 517.5 tok/s, about 14.9%. The gpt-oss-20b variant that skips sliding-window layers gains about 4.8%. Architecture and execution determine how bit-width savings reach system throughput.

The precision tradeoff differs too. GAAC adds 1.09–1.14× lossless compression to already quantized FP8 E4M3 and 1.25–1.30× to E5M2, preserving the quantized values. Task-quality tests still need to cover the initial FP8 approximation and the accumulation behavior of the consuming kernel.

Start the comparison from the deployed baseline. A system with mature FP8 kernels measures the incremental benefit of further compression. A system that needs original BF16 KV measures the cost of preserving it. Align hardware, architecture, precision, batch size, and workload, then compare.

Integration is where the work starts

GAAC's exponent windows depend on calibration. In the positional-drift experiment in Appendix C, a Qwen3-8B layer uses windows calibrated on prompts of at most 8K tokens to process 32K input. Overflow storage for later keys grows substantially; 32K calibration is more stable. RoPE can shift key-exponent distributions with position. Reconstruction remains lossless, but compression efficiency and fallback overhead change.

Cross-task tests provide a useful starting point: windows selected on WikiText retain similar compression across several code, math, and knowledge tasks. Next, run longer contexts and sustained request traces while tracking slot waste, fallback rates, prefix sharing, and cache turnover. Production workloads keep changing shape; the cache has to handle that.

On Apple M4, the authors implement compressed attention in Metal and compare it with a BF16-KV kernel of the same structure. At 64K context, gains are about 1.10× for one sequence and 1.25× for eight. The kernel has crossed a platform boundary. Putting it into a complete Mac inference framework and measuring first-token, ongoing-generation, and task-completion time is the next useful step.

Personal devices also have a thermal and battery budget. The paper provides memory and timing results; power and energy remain to be measured. Evaluating extra arithmetic against reduced traffic requires power traces, runtime, and sustained-load tests. Energy per token belongs in a local-deployment comparison too.

The comparisons I would run before deployment

Fix a reproducible baseline: model and quantization versions, inference framework and kernel, context and output lengths, KV precision, batch size, and request arrivals. Account for weights, resident KV, and workspace separately so the bottleneck has somewhere to show up.

Run two separate comparisons. Hold concurrency fixed and measure complete decode-step and request latency. Fix the memory budget, allow more concurrency, and measure aggregate throughput, queueing, and goodput. One asks how much faster the same user's request gets; the other asks how much more work the device can take.

Latency reporting needs its own distinctions. vLLM's metrics documentation separates time to first token (TTFT), inter-token latency (ITL), and average time per output token (TPOT). State whether queueing is included, how observations are aggregated, and which output-token denominator is used. An average tokens/s can conceal both a short request's wait and a long request's stalls. P50/P95/P99, task quality, sustained memory use, and energy measurements bring the comparison closer to actual use.

For agents, I would also save task traces containing tool waits and repeated prefill. If a task spends most of its duration waiting for external tools, improving decode by 10% will not cut the whole task by 10%. Finding where the time goes determines which layer is worth optimizing next.

The easiest mistake in local inference is treating a model that fits as a job finished. Smaller weights leave historical caches, data movement, and scheduling with their own bills. GAAC trades extra arithmetic for less traffic, and the gains are realized differently along the execution path. The next time a result promises 20% more speed, I will look for the complete timing breakdown before deciding to integrate it.