📖 For the best reading experience with properly rendered equations, view this article on GitHub Pages.

Mixtral has 46.7 billion parameters but only uses 12.9 billion per token.

It activates 2 of its 8 expert networks, leaving 72% of its weights untouched on every forward pass. This should be wasteful. Instead, it matches Llama 2 70B while running at the cost of a 13B model.

The trick is a property called sparsity.

The Property

A matrix is sparse if most of its elements are zero:

sparsity = number of zeros / total elements

A 90% sparse matrix has 9 zeros for every nonzero value.

Why should this help? Consider matrix-vector multiplication:

yᵢ = Σⱼ Wᵢⱼ xⱼ

If Wᵢⱼ = 0, that term contributes nothing. Skip it.

With 90% sparsity, you skip 90% of the multiplications. 10x speedup, right?

Not quite. There’s a catch.

The GPU Problem

Here’s the uncomfortable truth: GPUs don’t care about your zeros.

GPUs achieve speed through parallelism—thousands of cores executing the same operation on different data. They’re optimized for dense, regular memory access patterns.

Random sparsity breaks this:

• Memory access: Sparse formats store (row, column, value) tuples. Irregular access kills cache efficiency.

• Load balancing: Some rows might have 10 nonzeros, others 1000. Cores sit idle waiting for stragglers.

• Branching: Checking “is this zero?” adds overhead.

The result: you typically need >90% sparsity before unstructured sparse operations beat dense on a GPU. And even then, the gains are modest.

The math says skip. The hardware says “I don’t know how.”

Structured Sparsity: Making Hardware Happy

The solution: constrain your sparsity pattern so hardware can exploit it.

N:M Sparsity

NVIDIA’s Ampere architecture introduced hardware support for 2:4 sparsity: exactly 2 zeros in every group of 4 consecutive elements.

[a, 0, b, 0] [c, 0, 0, d] [0, e, f, 0] ...

This gives:

• 50% sparsity (2 of 4 are zero)

• 2x throughput on Tensor Cores

• Minimal accuracy loss (networks adapt during training)

The constraint is the feature: fixed positions mean predictable memory access, perfect load balancing, no branching.

Block Sparsity

Instead of individual zeros, zero out entire blocks:

W = | A   0   B |
    | 0   C   0 |
    | D   0   E |

Each block is either:

• Dense: Use optimized GEMM

• Zero: Skip entirely

Block sizes (32×32, 64×64) align with GPU tiles. The sparse indexing overhead amortizes across the block.

Weight Sparsity: Pruning

Neural networks are massively overparameterized. Many weights contribute almost nothing.

Magnitude pruning: Remove weights with smallest absolute values.

The surprising finding: you can often remove 90% of weights with minimal accuracy loss.

The Lottery Ticket Hypothesis

Frankle & Carlin (2019) showed something remarkable:

Dense networks contain sparse subnetworks (”winning tickets”) that—when trained in isolation from the same initialization—match the full network’s accuracy.

The interpretation: training a large network is really a search procedure. It finds the important subnetwork; the other weights were scaffolding.

Iterative Pruning

The practical recipe:

1. Train to convergence

2. Prune smallest 20% of weights

3. Fine-tune

4. Repeat until desired sparsity

Each round finds weights that seemed important but aren’t. Iterative pruning reaches higher sparsity than one-shot.

The Catch

Unstructured pruning creates random zeros—exactly what GPUs can’t exploit. Options:

• Accept the irregular memory access (works at very high sparsity)

• Use structured pruning (remove entire neurons, attention heads, layers)

• Prune to 2:4 patterns for hardware acceleration

Activation Sparsity: Free Zeros from ReLU

ReLU creates sparsity for free:

ReLU(x) = max(0, x)

Roughly half of activations become zero in typical networks. That’s 50% sparsity in every hidden layer.

But this sparsity is dynamic—you don’t know which values are zero until you compute them. GPUs still do the full computation, then throw away negatives.

Research directions:

• Predictive gating: Predict which neurons will be zero before computing them

• Top-k activation: Only keep the k largest activations

• Dynamic sparse training: Skip provably-zero computations

These are active research areas but not yet mainstream in production.

Mixture of Experts: Conditional Computation

Here’s the most elegant form of sparsity: don’t decide which weights are zero—decide which weights to use per input.

A Mixture of Experts layer:

y = Σᵢ G(x)ᵢ · Eᵢ(x)

where:

• E₁, E₂, …, Eₙ are N “expert” networks (typically MLPs)

• G(x) is a gating/routing function

• G(x) is sparse: only top-k experts get nonzero weights

With 8 experts and top-2 routing:

• Each token activates only 2 experts

• 75% of expert parameters are skipped per token

• But total model capacity is 8x larger

This is parameter sparsity at the architecture level.

Why MoE Works

Different inputs need different computations.

• “The cat sat on the mat” → language experts

• “∫ x² dx = x³/3 + C” → math experts

• deff∞():deff∞(): → code experts

Instead of one network that handles everything, you have specialists that activate on demand.

Research shows experts do specialize: different experts prefer different token types, topics, and patterns. The routing learns to match inputs to expertise.

MoE in Practice

Switch Transformer (Google, 2021): Simplified MoE with top-1 routing. Scaled to 1.6 trillion parameters.

Mixtral 8x7B (Mistral, 2023):

• 8 experts per layer

• Top-2 routing per token

• 46.7B total parameters

• 12.9B active per forward pass

• Matches Llama 2 70B at 5x less compute

GPT-4: Reportedly uses MoE architecture.

Gemini: Uses MoE.

The pattern is clear: frontier models use conditional computation.

Sparse Attention

Full attention is O(n²) for sequence length n. At 100K tokens, that’s 10 billion attention computations per layer.

Sparse attention reduces this by only attending to a subset of positions.

Sliding Window

Each position attends only to nearby positions:

Position i attends to: [i-w, ..., i-1, i, i+1, ..., i+w]

Complexity: O(n · w) where w is window size.

This works because most relevant context is local. Long-range dependencies exist but are rarer.

Strided Patterns

Add periodic global attention:

Position i attends to:
- Local: [i-w, ..., i+w]
- Global: [0, k, 2k, 3k, ...] for some stride k

Captures both local detail and global structure.

The Reality Check

FlashAttention changed the calculus. By making exact attention memory-efficient, it reduced pressure for approximate sparse methods.

Modern LLMs typically use:

• Short context: Full attention with FlashAttention

• Long context: Sliding window + some global tokens

• Very long context: Hierarchical or chunked approaches

Sparse attention is less about compute savings and more about memory constraints at extreme sequence lengths.

The Unifying Principle

All forms of sparsity share one insight:

Not all computation is equally valuable. Skip the parts that don’t matter.

The challenge is always: turning mathematical sparsity into actual speedup.

The solution is always: structure. Align your sparsity with hardware.

Why Sparsity Works At All

Why can you remove 90% of weights without destroying the model?

Overparameterization: Neural networks have far more parameters than strictly necessary. This excess provides:

• Easier optimization (smoother loss landscape)

• Implicit regularization

• Robustness to initialization

Once trained, much of this capacity is redundant.

The Lottery Ticket View: Training is search. The dense network explores many possible subnetworks. Most are dead ends; one is the solution. Pruning removes the dead ends.

The MoE View: Different inputs need different parameters. A dense network wastes capacity using the same weights for all inputs. Experts specialize, using total capacity more efficiently.

The Takeaway

Sparsity is the mathematical license to skip computation.

If Wᵢⱼ = 0 or unused, then skip Wᵢⱼ · xⱼ

But hardware doesn’t exploit random zeros. Unstructured sparsity gives unstructured memory access, which GPUs hate.

The practical path:

• 2:4 structured sparsity: 2x speedup with hardware support

• Block sparsity: Dense blocks, sparse selection

• Mixture of Experts: Let the input decide what to compute

The most powerful form isn’t pruning—it’s conditional computation. MoE achieves massive parameter counts with modest compute by activating only what’s needed.

This is why Mixtral matches models 5x its active compute. Having idle parameters isn’t waste—it’s the whole point. Different inputs use different parameters. No single forward pass needs them all.

Sparsity says: you don’t have to compute everything.
Structure says: here’s how to actually skip it.
MoE says: let the input decide what matters.

Previous article: Domain Transformations: The Art of Finding Easier Spaces

Further Reading

• Frankle & Carlin, “The Lottery Ticket Hypothesis” (2019) — Sparse subnetworks from initialization

• Fedus et al., “Switch Transformers” (2021) — Scaling MoE to trillions of parameters

• NVIDIA, “Accelerating Sparse Deep Neural Networks” (2021) — 2:4 structured sparsity on Ampere

• Jiang et al., “Mixtral of Experts” (2024) — State-of-the-art open MoE

• Pool & Yu, “Accelerating Transformer Pre-training with 2:4 Sparsity” (2024) — Training with structured sparsity

📖 For the best reading experience with properly rendered equations, view this article on GitHub Pages.

Here’s a trick that appears everywhere in numerical computing:

Don’t solve the hard problem. Transform it into an easy one.

Multiplying a thousand small probabilities? You’ll underflow to zero. But add their logarithms, and everything works.

Convolving two signals of length n? That’s O(n²). But multiply in Fourier space, and it’s O(n log n).

These aren’t clever hacks. They’re instances of a general principle: the right domain makes computation tractable.

The Principle

A domain transformation moves computation from one space to another:

Hard problem in Space A → transform → Easy problem in Space B

The pattern:

1. Transform input to the easier domain

2. Compute in that domain

3. Transform back (if needed)

This is worthwhile when:

cost(transform) + cost(easy computation) + cost(inverse) < cost(hard computation)

Often dramatically so.

Logarithms: Multiplication to Addition

The most common domain transformation in ML.

The Problem

You’re computing the probability of a sequence:

P(x₁, x₂, …, xₙ) = ∏ᵢ P(xᵢ | x<ᵢ)

Each factor is less than 1. Multiply 1000 of them:

>>> 0.1 ** 1000
0.0  # Underflow

The true answer isn’t zero—it’s just smaller than the smallest representable float.

The Solution

Work in log-space:

log P = Σᵢ log P(xᵢ | x<ᵢ)

Multiplication becomes addition. Products of tiny numbers become sums of negative numbers.

>>> import math
>>> 1000 * math.log(0.1)
-2302.58...  # No underflo

This is why every language model outputs log-probabilities. Why Bayesian inference uses log-likelihoods. Why HMMs compute in log-space.

Logarithms don’t just prevent underflow. They turn an unstable operation into a stable one.

The Log-Sum-Exp Trick

But what if you need to add probabilities, not multiply them?

For mutually exclusive events:

P(A or B) = P(A) + P(B)

In log-space, this becomes:

log(eᵃ + eᵇ)

where a = log P(A) and b = log P(B). This comes up constantly—marginalizing over hidden states in HMMs, summing over paths in CTC, computing partition functions.

The Problem

>>> import math
>>> a, b = -1000, -1001
>>> math.log(math.exp(a) + math.exp(b))
# math.exp(-1000) = 0.0 (underflow)
# Result: undefined

Exponentiating large negative numbers underflows. We’re back where we started.

The Solution

Factor out the maximum:

log(eᵃ + eᵇ) = max(a, b) + log(1 + e^(−|a−b|))

The exponent −|a−b| is always ≤ 0, so e^(−|a−b|) ≤ 1. No overflow, and the underflowed term is added to 1, so we don’t lose precision.

def log_sum_exp(a, b):
    max_val = max(a, b)
    return max_val + math.log(math.exp(a - max_val) + math.exp(b - max_val))

For vectors:

logsumexp(x) = max(x) + log Σᵢ exp(xᵢ − max(x))

This is exactly what softmax uses internally:

softmax(x)ᵢ = exp(xᵢ − logsumexp(x))

The log-sum-exp trick keeps computation in log-space while handling addition.

Fourier Transform: Convolution to Multiplication

Another profound domain transformation.

The Problem

Convolution in the spatial domain:

(f ∗ g)[n] = Σₘ f[m] · g[n−m]

For signals of length N, this is O(N²)—N outputs, each summing over N terms.

The Convolution Theorem

The Fourier transform has a remarkable property:

ℱ(f ∗ g) = ℱ(f) · ℱ(g)

Convolution in space = multiplication in frequency.

The algorithm:

def fast_convolve(f, g):
    F_f = fft(f)        # O(n log n)
    F_g = fft(g)        # O(n log n)
    F_result = F_f * F_g  # O(n) - element-wise
    return ifft(F_result) # O(n log n)

Total: O(n log n) instead of O(n²).

For large kernels, the transform cost is negligible. For small kernels (3×3, 5×5), direct convolution is faster—the transform overhead dominates.

In practice, modern deep learning frameworks use highly optimized direct convolution (cuDNN) even for medium-sized kernels, because GPU implementations are so fast. But FFT-based convolution remains important in signal processing and for very large kernels.

Why Transformers Don’t Use FFT

Attention looks like it should benefit from FFT:

Attention = softmax(QKᵀ)V

The QKᵀ multiplication is O(n²d) for sequence length n.

But there’s a catch: attention isn’t convolution.

Convolution is translation-equivariant—shift the input, shift the output. The kernel slides uniformly.

Attention is content-based—each position attends differently based on what’s there, not where it is.

The Fourier transform exploits the structure of convolution. Attention doesn’t have that structure.

(Some efficient attention methods like Performers and Linear Attention do use kernel approximations—but they’re approximating attention, not computing it exactly via FFT.)

Embeddings: Sparse to Dense

A different kind of domain transformation.

The Problem

Words are categorical—there’s no natural arithmetic on them.

“king” - “man” + “woman” = ???

With one-hot encoding, words are orthogonal vectors in a 50,000-dimensional space. No similarity, no structure.

The Solution

Learn a dense embedding:

word → embed → ℝᵈ

Now:

embed(”king”) − embed(”man”) + embed(”woman”) ≈ embed(”queen”)

The transformation creates structure that wasn’t there before.

This isn’t just dimensionality reduction. It’s finding a space where the hard problem (word relationships) becomes easy (vector arithmetic).

The Kernel Trick: Linear in a Lifted Space

Sometimes the solution isn’t a lower-dimensional space—it’s a higher one.

The Problem

Data isn’t linearly separable:

Copy

         ○   ○   ○
       ○           ○
     ○    ●  ●  ●    ○
     ○    ●  ●  ●    ○
     ○    ●  ●  ●    ○
       ○           ○
         ○   ○   ○

  No linear boundary separates ● from ○.

The inner class is surrounded. No line can divide them.

The Solution

Map to a higher-dimensional space where it is linearly separable:

φ: ℝᵈ → ℝᴰ where D > d

For the concentric pattern, add just one feature: z = x² + y² (squared distance from center). Now the inner class has small z, the outer class has large z. A horizontal plane separates them.

The kernel trick computes dot products in the lifted space without explicitly computing the transformation:

K(x, y) = ⟨φ(x), φ(y)⟩

For the RBF kernel:

K(x, y) = exp(−‖x − y‖² / 2σ²)

This implicitly works in an infinite-dimensional space, but you never compute it directly.

The kernel trick is a domain transformation you don’t have to pay for.

Spectral Graph Methods

Graph Laplacians transform graph problems into linear algebra.

The Problem

Clustering nodes in a graph based on connectivity. Community detection. Graph partitioning.

These seem like discrete, combinatorial problems.

The Solution

Compute the graph Laplacian:

L = D − A

where D is the degree matrix and A is the adjacency matrix.

The eigenvectors of L embed nodes into Euclidean space:

nodeᵢ → (v₁(i), v₂(i), …, vₖ(i))

In this space, k-means clustering solves the graph partitioning problem.

A discrete optimization problem becomes a tractable eigenvector computation.

Numerical Stability: A Recurring Theme

Many domain transformations exist for numerical reasons:

Each is a domain transformation that trades mathematical equivalence for numerical stability.

Equivalent in exact arithmetic. Different in floating-point.

When to Look for a Transform

The pattern recurs:

1. Multiplication → Addition: Use logarithms

2. Convolution → Element-wise: Use Fourier

3. Non-linear → Linear: Use kernel methods

4. Discrete → Continuous: Use spectral methods

5. High-dimensional sparse → Low-dimensional dense: Use embeddings

The skill is recognizing when your hard problem has an easier dual.

Questions to ask:

• What’s expensive or unstable? Products of small numbers? Convolution? Non-linear optimization?

• Is there a standard transform? Log, Fourier, Laplace, embeddings?

• Does the transform preserve what you need? Some transforms lose information.

• Is the round-trip worth it? Transform cost + easy computation < hard computation?

The Meta-Lesson

The previous articles in this series covered:

• Associativity: Enables chunking, parallelization, streaming

• Commutativity: Enables reordering, permutation invariance

• Linearity: Enables batching, gradient accumulation

Domain transformations are different. They’re not about properties of operations—they’re about choosing the right space.

But the connection is deep: transforms often work because they reveal simpler structure.

• Fourier reveals that convolution is pointwise in frequency space

• Logarithms reveal that multiplication is additive in log space

• Embeddings reveal that semantic relationships are approximately linear in embedding space

The transform finds a space where the problem has the structure you need.

The Takeaway

Don’t solve the hard problem directly. Ask: is there an easier space?

• Products underflow → work in log space

• Convolution is O(n²) → work in frequency space

• Data isn’t linearly separable → work in kernel space

The transform might cost something. But if the computation in the new space is dramatically cheaper, you win.

This is computational judo: using structure to redirect effort. The best algorithms don’t fight the problem—they find the space where the problem solves itself.

See also:

• The One Property That Makes FlashAttention Possible — Associativity

• Why Transformers Need Positional Encodings — Commutativity

• Why Batching Works — Linearity

Further Reading

• Numerical Recipes, Ch. 12-13 — FFT and spectral methods

• Murphy, “Probabilistic Machine Learning” — Log-space computation throughout

• Scholkopf & Smola, “Learning with Kernels” — Kernel methods and feature spaces

• Von Luxburg, “A Tutorial on Spectral Clustering” — Graph Laplacians and embeddings

One of his key heuristics: “Do you know a related problem? Here is a problem related to yours and solved before. Could you use it?”

In the 1990s, Japanese engineer Eiji Nakatsu faced a difficult challenge: the Shinkansen bullet train created deafening sonic booms when entering tunnels at high speed. The solution came not from aerospace engineering, but from birdwatching. Nakatsu noticed that kingfishers dive from air into water with barely a splash—a transition between mediums, just like a train entering a tunnel. By reshaping the train’s nose to mimic the kingfisher’s beak, he eliminated the sonic boom and improved efficiency.

This article applies that same heuristic to a classic distributed systems challenge: generating unique IDs at scale.

The standard answers—UUID, database sequences, Snowflake—are well-known. But there’s a deeper design space that most discussions miss. To find it, we’ll ask Polya’s question: What related problem has been solved before?

The answer, it turns out, has been shipping with your operating system for decades.

Step 1: Understand the Problem

Polya’s first step is to understand the problem completely before attempting solutions. What is the unknown? What are the constraints?

What Is The Unknown?

We need a function that returns identifiers with this property: no two calls, anywhere in a distributed system, ever return the same value.

What Are The Constraints?

Every ID generation system makes trade-offs across these dimensions:

ConstraintQuestionUniquenessCan two nodes ever generate the same ID?OrderingDo IDs reflect temporal or causal order?SizeHow many bits? (Affects storage, indexing, transmission)ThroughputHow many IDs per second per node?CoordinationHow much communication between nodes?AvailabilityCan we generate IDs during network partitions?

Most systems need uniqueness. Everything else is negotiable—but every choice has consequences.

Have We Seen This Problem Before?

The standard solutions are well-documented:

UUIDs offer zero coordination—any node generates 128 random bits independently. But they’re large, unsortable, and their randomness destroys B-tree locality. Every insert lands on a random leaf page.

Database auto-increment gives you perfect ordering and minimal size, but requires a round-trip to a central database for every ID. It’s a throughput bottleneck and a single point of failure.

Twitter’s Snowflake hit a sweet spot: 64 bits, time-sortable, high throughput, and minimal coordination (just a one-time machine ID assignment). Its structure—41 bits of timestamp, 10 bits of machine ID, 12 bits of sequence—became the template everyone copies.

┌─────────────────────────────────────────────────────────────┐
│                    Snowflake ID (64 bits)                   │
├─────────┬───────────────────┬─────────────┬─────────────────┤
│ 1 bit   │ 41 bits           │ 10 bits     │ 12 bits         │
│ (sign)  │ (timestamp ms)    │ (machine)   │ (sequence)      │
├─────────┴───────────────────┴─────────────┴─────────────────┤
│ ~69 years │ 1024 machines │ 4096 IDs/ms/machine             │
└─────────────────────────────────────────────────────────────┘

Snowflake is excellent. But is it the only answer? Is there a deeper pattern we’re missing?

This is where most discussions stop. We won’t.

Step 2: Devise a Plan — Finding the Related Problem

Polya’s second step is to devise a plan. His most powerful heuristic here: “Do you know a related problem?”

Let’s think about what we’re really doing. We need to:

• Hand out unique resources (IDs)

• To concurrent requesters (distributed services)

• With minimal coordination (for performance)

Where else have we seen this exact structure?

The Leap: Memory Allocation

Memory allocators have been solving a structurally identical problem since the 1980s:

• Hand out unique resources (memory addresses)

• To concurrent requesters (threads)

• With minimal coordination (for performance)

The mapping is direct:

ID GenerationMemory AllocationUnique IDUnique memory addressDistributed serviceThreadCentral coordinatorGlobal heapNetwork round-tripLock acquisition

Could we use this? Let’s look at how memory allocators solved it.

TCMalloc’s Solution: Hierarchical Allocation

Google’s tcmalloc uses a three-tier hierarchy:

┌─────────────────────────────────────────────────────────────┐
│                         tcmalloc                            │
│                                                             │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐          │
│  │Thread Cache │  │Thread Cache │  │Thread Cache │  ...     │
│  │  (no locks) │  │  (no locks) │  │  (no locks) │          │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘          │
│         │                │                │                 │
│         └────────────────┼────────────────┘                 │
│                          ▼                                  │
│                  ┌───────────────┐                          │
│                  │ Central Free  │                          │
│                  │    Lists      │  ← One lock per          │
│                  │ (per size)    │    size class            │
│                  └───────┬───────┘                          │
│                          │                                  │
│                          ▼                                  │
│                  ┌───────────────┐                          │
│                  │  Page Heap    │  ← Global, but           │
│                  │               │    rarely touched        │
│                  └───────────────┘                          │
│                                                             │
└─────────────────────────────────────────────────────────────┘

The key insight: push work to the edges, coordinate only when necessary.

• Thread caches handle most allocations with zero synchronization. Each thread has its own pool of objects.

• Central lists refill thread caches when they’re empty. This requires locks, but happens infrequently.

• Page heap provides bulk memory to central lists. Even more infrequent.

The documentation states it plainly: “when following this fast path, TCMalloc acquires no locks at all.”

This is amortized coordination. You pay the cost of synchronization once per N allocations, where N is your cache size. For memory allocators, N might be hundreds of objects. For ID generators, N can be thousands.

Extracting the Transferable Principle

The pattern generalizes beyond memory:

Memory AllocatorID AllocatorThread cacheLocal ID bufferArena/Central listRegional allocatorPage heap/mmapRoot coordinatorSize classID rangeObjectIndividual ID

We’ve found our related problem. Now let’s apply it.

Step 3: Carry Out the Plan — Applying the Pattern

With the memory allocator insight in hand, we can derive multiple solutions by applying the same principles.

Solution 1: Slab Allocation

Instead of generating IDs one at a time (Snowflake) or from a central source (database), we pre-allocate ranges:

┌─────────────────────────────────────────────────────────────┐
│                    Slab-Based ID Allocation                 │
│                                                             │
│   Central Database:                                         │
│   ┌─────────────────────────────────────────────────────┐   │
│   │ business_tag │ max_allocated │ step_size │          │   │
│   │ orders       │ 50,000        │ 10,000    │          │   │
│   │ users        │ 1,000,000     │ 50,000    │          │   │
│   └─────────────────────────────────────────────────────┘   │
│                          │                                  │
│              Request: “Give me a range”                     │
│              Response: [50000, 60000)                       │
│                          │                                  │
│         ┌────────────────┼────────────────┐                 │
│         ▼                ▼                ▼                 │
│   ┌───────────┐    ┌───────────┐    ┌───────────┐           │
│   │ Service A │    │ Service B │    │ Service C │           │
│   │[50K, 60K) │    │[60K, 70K) │    │[70K, 80K) │           │
│   │ pos: 50847│    │ pos: 62103│    │ pos: 70002│           │
│   └───────────┘    └───────────┘    └───────────┘           │
│                                                             │
│   Local allocation: just increment a counter                │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Each service instance:

1. Requests a range from the central database (one network call)

2. Hands out IDs from that range locally (zero network calls, just counter++counter++)

3. When exhausted, requests another range

If your range size is 10,000, you’ve reduced coordination by 10,000×.

Production Example: Meituan Leaf

Meituan, China’s largest food delivery platform, open-sourced their ID generator called Leaf. Its “segment mode” implements exactly this pattern, with a crucial optimization: double buffering.

┌─────────────────────────────────────────────────────────────┐
│                    Leaf Service Instance                    │
│                                                             │
│   ┌─────────────────────┐    ┌─────────────────────┐        │
│   │     Segment A       │    │     Segment B       │        │
│   │   [100000, 110000)  │    │   [110000, 120000)  │        │
│   │   current: 108,934  │    │   (ready, waiting)  │        │
│   │   ███████████░░░░   │    │   ░░░░░░░░░░░░░░░░  │        │
│   │      ACTIVE         │    │      STANDBY        │        │
│   └─────────────────────┘    └─────────────────────┘        │
│                                                             │
│   When Segment A reaches 10% remaining:                     │
│     → Background thread fetches next range into Segment B   │
│                                                             │
│   When Segment A exhausted:                                 │
│     → Atomic switch to Segment B (already loaded!)          │
│     → Background thread now loads Segment A                 │
│                                                             │
└─────────────────────────────────────────────────────────────┘

This eliminates latency spikes at range boundaries. The prefetching is triggered proactively—Leaf starts loading the next segment when the current one drops below 10% capacity.

Sound familiar? It’s exactly how CPU prefetching works. Or how a streaming service buffers the next video segment while you watch the current one. Or how jemalloc returns memory to central lists before completely exhausting local caches.

The principle is universal: anticipate needs, prefetch resources, smooth out latency.

Trade-offs

Wins:

• Near-zero coordination cost (amortized to 1/N)

• Sequential IDs within each slab (excellent B-tree locality)

• No clock dependency (pure counters)

• Simple mental model

Loses:

• No global time ordering (slab A might be issued before slab B, but contain higher IDs)

• Wasted IDs on crash (uncommitted portion of slab is lost)

• Central coordinator dependency (though it’s touched rarely)

Solution 2: Hierarchical Allocation

What if the central coordinator becomes a bottleneck? Or a single point of failure?

Here’s where we can go deeper than existing systems. Memory allocators don’t just have two tiers—tcmalloc has three (thread cache → central → page heap), and the kernel has more below that (page allocator → buddy system → physical memory).

We can apply the same hierarchy to ID allocation.

┌─────────────────────────────────────────────────────────────┐
│                  Hierarchical ID Allocation                 │
│                                                             │
│                    ┌─────────────────┐                      │
│                    │ Root Allocator  │                      │
│                    │  [0, 2^64)      │                      │
│                    │ Grants: 2^40    │                      │
│                    └────────┬────────┘                      │
│                             │                               │
│          ┌──────────────────┼──────────────────┐            │
│          ▼                  ▼                  ▼            │
│   ┌─────────────┐    ┌─────────────┐    ┌─────────────┐     │
│   │ Regional US │    │ Regional EU │    │Regional APAC│     │
│   │ [0, 2^40)   │    │[2^40, 2^41) │    │[2^41,3×2^40)│     │
│   │ Grants: 2^24│    │ Grants: 2^24│    │ Grants: 2^24│     │
│   └──────┬──────┘    └──────┬──────┘    └──────┬──────┘     │
│          │                  │                  │            │
│    ┌─────┼─────┐           ...                ...           │
│    ▼     ▼     ▼                                            │
│  ┌───┐ ┌───┐ ┌───┐                                          │
│  │L1 │ │L2 │ │L3 │   Leaf nodes (app instances)             │
│  └───┘ └───┘ └───┘   Grants: 2^12 (4096 IDs)                │
│                                                             │
└─────────────────────────────────────────────────────────────┘

This mirrors exactly how ma≪ocma≪oc works:

• Your application calls ma≪oc()ma≪oc() → thread-local fast path

• Thread cache exhausted → refill from arena

• Arena exhausted → mmap()mmap() or sbrk()sbrk() to kernel

• Kernel → physical page allocator

Each tier handles progressively larger allocations at progressively lower frequencies.

Request Frequency Analysis

Let’s do the math. Assume:

• Leaf nodes get 4,096 IDs per range (2^12)

• Regional nodes get 16 million IDs per range (2^24)

• Root holds the full 64-bit space

If your system generates 1 million IDs per second globally:

The root allocator is essentially write-once infrastructure. You could run it on a napkin.

Failure Isolation

This hierarchy provides graceful degradation:

Root dies:

• Regionals continue with their pre-allocated ranges

• At 2^40 IDs per regional, that’s years of runway

• Plenty of time to restore from backup

Regional dies:

• Leaf nodes continue with their local buffers

• Leaves can failover to sibling regional

• Or escalate directly to root (emergency path)

Leaf dies:

• Only loses its local buffer (4K IDs max)

• Other leaves completely unaffected

• Restart gets fresh range from regional

Compare this to Snowflake, where a clock going backward can corrupt your entire ID space, or to flat slab allocation, where the central database is touched by every node.

Adaptive Chunk Sizing

Just as modern allocators adjust allocation sizes based on demand, hierarchical ID allocators can adapt:

class AdaptiveAllocator:
    def __init__(self, tier, parent):
        self.tier = tier
        self.parent = parent
        self.base_chunk = BASE_SIZES[tier]
        self.current_chunk = self.base_chunk

    def request_range(self, urgency=”normal”):
        # Track consumption rate
        consumption_rate = self.ids_used / self.time_elapsed

        if urgency == “emergency” or consumption_rate > THRESHOLD:
            # We’re burning through IDs fast, request more
            self.current_chunk = min(
                self.current_chunk * 2,
                self.base_chunk * 16
            )
        elif consumption_rate < THRESHOLD / 10:
            # We’re barely using IDs, request less
            self.current_chunk = max(
                self.current_chunk / 2,
                self.base_chunk / 4
            )

        return self.parent.allocate(self.current_chunk)

A sudden traffic spike? Leaves request larger chunks. Traffic dies down? Request smaller chunks to avoid waste. The system breathes with demand.

The Buddy System Angle

For even more sophisticated range management, consider the buddy allocator pattern:

Initial space: [0, 2^32)

Split on demand (power-of-2 blocks):

                    [0, 2^32)
                   /         \
           [0, 2^31)        [2^31, 2^32)
           /      \              ...
    [0, 2^30)  [2^30, 2^31)
       ...         ...

Benefits:
- Easy to find appropriately-sized blocks
- Simple coalescing when ranges returned
- Enables dynamic rebalancing

When a service scales down and returns unused IDs, buddy coalescing can reclaim and redistribute them. This is overkill for most systems, but demonstrates the depth of the design space.

Solution 3: Consensus as ID Generation

Now for a different approach entirely.

What if you already run a consensus system—ZooKeeper, etcd, a Raft-based database? You’re already paying for consensus. Can you get IDs for free?

The Log Index Is Your ID

In Multi-Paxos or Raft, every committed entry has a unique, monotonically increasing log index. This index is:

• Unique (by definition of consensus)

• Totally ordered (slot N comes before slot N+1)

• Durable (replicated to a quorum)

• Agreed upon (that’s what consensus means)

That’s... an ID.

┌─────────────────────────────────────────────────────────────┐
│                 Consensus Log as ID Generator               │
│                                                             │
│   Log Index:  ... | 47 | 48 | 49 | 50 | 51 | ...           │
│   Content:        |cmd |cmd |cmd |cmd |cmd |               │
│                                                             │
│   Each index IS a unique, ordered identifier                │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Real Implementations

ZooKeeper’s Sequential Znodes:

zk.create(”/ids/id-”, data, CreateMode.PERSISTENT_SEQUENTIAL);
// Returns: “/ids/id-0000000047”
// The suffix is the consensus sequence number

etcd’s Revision Number:
Every write to etcd increments a global revision. With proper batching, etcd achieves 60,000-180,000 operations per second on modern hardware.

Neon’s Log Sequence Numbers:
Neon (the serverless Postgres company) uses a Paxos-like protocol for WAL sequence numbers. Their compute nodes propose entries, safekeepers agree on ordering, and the consensus slot becomes the LSN.

Batching and Pipelining

Naive consensus-per-ID is slow (one round-trip per ID). But you can batch:

Time window (1ms):
  Request 1: “need an ID”
  Request 2: “need an ID”
  ...
  Request 100: “need an ID”

Leader batches into single consensus round:
  Propose: “Allocate IDs [1000, 1100)”

After commit:
  Request 1 → 1000
  Request 2 → 1001
  ...
  Request 100 → 1099

Amortized: 1 consensus round per 100 IDs

You can also pipeline—propose slot N+1 before slot N commits:

t=0: Propose slot 100 (IDs 10000-10099)
t=1: Propose slot 101 (IDs 10100-10199)  ← Don’t wait!
t=2: Propose slot 102 (IDs 10200-10299)
t=3: Commit for slot 100 arrives
t=4: Commit for slot 101 arrives
...

With batching (1000 IDs per round), pipelining (depth 4), and 2ms round-trip:

Throughput = (1000 × 4) / 0.002s = 2,000,000 IDs/sec

That’s competitive with Snowflake.

When Consensus Wins

You already have it: If you’re running etcd, ZooKeeper, or a Raft-based database, ID generation is nearly free.

Strict global ordering: Unlike Snowflake (where ID 43 can be created after ID 137 across nodes), consensus gives you:

ID ordering = causal ordering

Clock skepticism: Edge computing, IoT, multi-cloud with NTP variance—anywhere clocks are unreliable.

Transactional semantics:

BEGIN;
  INSERT INTO orders (id, ...) VALUES (next_id(), ...);
  -- If transaction aborts, ID is never committed
  -- Can be reclaimed (unlike Snowflake)
COMMIT;

When Consensus Loses

Cross-region latency: 70ms round-trip between US and EU means minimum 70ms per ID (without heavy batching). Snowflake: 0ms.

Availability during partitions: Consensus requires quorum. Network partition → minority partition can’t generate IDs. Snowflake works independently.

Simplicity: Snowflake is ~50 lines. Production Raft is ~10,000+.

Solution 4: Hybrid Approaches

The most powerful designs combine these approaches.

Consensus-Backed Slab Allocation

Use consensus to allocate ranges, then bump-pointer locally:

┌─────────────────────────────────────────────────────────────┐
│              Consensus-Backed Slab Allocator                │
│                                                             │
│                    ┌─────────────────┐                      │
│                    │  Raft/Paxos     │                      │
│                    │  Cluster (3-5)  │                      │
│                    └────────┬────────┘                      │
│                             │                               │
│           Consensus on: “Service A gets [1M, 2M)”           │
│                             │                               │
│         ┌───────────────────┼───────────────────┐           │
│         ▼                   ▼                   ▼           │
│    ┌─────────┐         ┌─────────┐         ┌─────────┐      │
│    │Service A│         │Service B│         │Service C│      │
│    │[1M, 2M) │         │[2M, 3M) │         │[3M, 4M) │      │
│    └─────────┘         └─────────┘         └─────────┘      │
│                                                             │
│    Local: counter++ (zero coordination)                     │
│    Range exhausted: back to consensus cluster               │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Benefits:

• Range allocations are durable and consistent (consensus)

• Individual IDs are fast (local counter)

• Full audit trail (”who got what range when”)

• Clean failure recovery (replay consensus log)

Snowflake + Slab Hybrid

Embed slab allocation in the Snowflake structure:

┌─────────────────────────────────────────────────────────────┐
│                  Hybrid ID (64 bits)                        │
├─────────────────┬─────────────────┬─────────────────────────┤
│ 32 bits         │ 16 bits         │ 16 bits                 │
│ Slab number     │ Timestamp       │ Sequence                │
│ (from central)  │ (seconds)       │ (local)                 │
├─────────────────┴─────────────────┴─────────────────────────┤
│ - Slabs provide rough ordering and locality                 │
│ - Timestamp within slab maintains time ordering             │
│ - Sequence handles burst within same second                 │
└─────────────────────────────────────────────────────────────┘

You get slab locality for database writes AND time ordering within slabs.

Step 4: Look Back — The Universal Pattern

Polya’s final step: “Can you use the result or method for other problems?”

We found that memory allocators solved our problem decades ago. But the pattern is even more universal. The same structure appears across computer science:

Filesystems (ext4)

ext4 divides disk into block groups, each with its own bitmap:

┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Block Group 0│ │ Block Group 1│ │ Block Group 2│
│ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │
│ │ Bitmap   │ │ │ │ Bitmap   │ │ │ │ Bitmap   │ │
│ │ Inodes   │ │ │ │ Inodes   │ │ │ │ Inodes   │ │
│ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │
└──────────────┘ └──────────────┘ └──────────────┘

Local allocation within groups, minimal global contention. Same pattern.

Streaming (Kafka)

Kafka partitions have independent offset sequences:

Partition 0: [0, 1, 2, 3, 4, ...]  ← Sequential within
Partition 1: [0, 1, 2, 3, ...]     ← Independent sequences
Partition 2: [0, 1, 2, ...]        ← No cross-partition ordering

Same pattern: per-partition sequencing, no global coordination.

Scheduling (Work Stealing)

Parallel schedulers give each worker a local queue. When empty, steal from others:

Worker A: [████]     Worker B: [    ] ← steal → Worker C: [████████]

Same pattern: local resources, coordinate only when necessary.

Time (Hybrid Logical Clocks)

CockroachDB’s HLC combines physical and logical time:

┌─────────────────────────┬───────────────────────────┐
│ Physical (wall clock)   │ Logical (counter)         │
└─────────────────────────┴───────────────────────────┘

Stays close to real time, but logical component ensures causality. Hybrid approach—same principle.

The Meta-Pattern

All these domains solve the same fundamental problem: distributing access to a shared resource while minimizing coordination.

The solution is always some variant of:

1. Hierarchy — Multiple tiers with different granularities

2. Locality — Push work to the edges

3. Batching — Amortize coordination costs

4. Prefetching — Anticipate needs before they arise

These patterns were discovered in OS research in the 1970s-80s. They were refined in memory allocators through the 90s-2000s. And they apply directly to distributed systems today.

Why the Patterns Transfer: Parallel ⊂ Distributed

There’s a deeper reason why memory allocator patterns work for distributed ID generation: parallel programming is a special case of distributed programming.

Both solve the same fundamental problem: independent execution units coordinating access to shared resources. The difference is in the constraints:

The implication: If you solve a problem under distributed constraints, the solution automatically works for parallel. But not always vice versa.

This is why tcmalloc’s patterns transfer so cleanly—the coordination avoidance principles that work across threads (where lock acquisition is expensive) also work across continents (where network round-trips are expensive). The cost function changes, but the optimization strategy remains the same.

What transfers directly:

What doesn’t transfer (the “distributed tax”):

• Partial failure: In parallel systems, if one thread crashes, the process dies. Clean slate. In distributed systems, Node A crashes while B and C continue—with stale views of A. This is why distributed systems need failure detectors, consensus, and replication.

• Unbounded latency: The FLP impossibility result proves that deterministic consensus is impossible in asynchronous systems with even one crash failure. This fundamental limit doesn’t exist in parallel programming where timing is bounded.

• No shared memory: Parallel systems get cache coherency protocols (MESI) for free. Distributed systems must build replication and consistency from scratch.

The connecting figure: Leslie Lamport. His 1978 paper “Time, Clocks, and the Ordering of Events in a Distributed System“ introduced “happens-before”—a concept that applies equally to memory models (parallel) and causality (distributed). His bakery algorithm solved mutual exclusion for threads; Paxos solved consensus for nodes. Same mind, same patterns, different constraints.

Understanding this relationship is a superpower. When you face a distributed systems problem, ask: “How did parallel programming solve this?” Often, the core insight transfers—you just need to handle partial failure and unbounded latency on top.

Decision Framework

When should you use each approach?

Trade-off Summary

Conclusion: The Method, Not Just The Answer

We started with a question: how do you generate unique IDs in a distributed system?

The standard answer is Snowflake. But by asking Polya’s question—“Do you know a related problem?”—we discovered an entire design space hidden behind one well-known solution.

Memory allocators solved this problem decades ago. The same patterns appear in filesystems, streaming systems, schedulers, and distributed clocks. Once you see the connection, the solutions almost derive themselves.

The deeper lesson isn’t about ID generation. It’s about problem-solving.

Next time you face a hard distributed systems problem, ask:

1. What is the underlying structure? (Unique resources, concurrent access, coordination cost)

2. What other domain solved something similar? (OS, networking, databases, hardware)

3. Can I extract and transfer the principle? (Hierarchy, locality, batching, prefetching)

The best engineers don’t memorize solutions. They recognize patterns.

Eiji Nakatsu saw a kingfisher dive into water and redesigned a bullet train. The principles of tcmalloc power your ID generator. The patterns are everywhere—if you know how to look.

Further Reading

The Pattern

• TCMalloc Design — The memory allocator that inspired this thinking

• Mimalloc Paper — Free list sharding and segment allocation

The Solutions

• Meituan Leaf — Production slab-based ID generator

• Snowflake — Twitter’s original design

• Sonyflake — Variant optimized for longevity

• Neon’s Paxos — Consensus for log sequence numbers

• TiDB TSO — Timestamp oracle design

• Spanner Sequences — Batch allocation in practice

The Parallel ↔ Distributed Connection

• Time, Clocks, and the Ordering of Events — Lamport’s foundational paper explained

• FLP Impossibility — Why distributed is fundamentally harder

• Coordination Avoidance in Databases — Peter Bailis on when coordination is necessary

The Method

• How to Solve It — Polya’s classic on problem-solving heuristics

• The Shinkansen and Kingfisher — Cross-domain problem solving in action

📖 For the best reading experience with properly rendered equations, view this article on GitHub Pages.

Here’s something that should seem stranger than it does:

You can train a neural network on 1,000 samples almost as fast as on 10.

Not 100x slower. Almost the same speed.

How? The answer is a single property: linearity.

The Property

A function f is linear if:

f(αx + βy) = αf(x) + βf(y)

Scale the input, scale the output. Add inputs, add outputs. Combinations work as expected.

Matrix multiplication is the canonical linear operation:

f(x) = xW

Check: (αx + βy)W = αxW + βyW. ✓

This simple property is why modern deep learning is computationally tractable.

Why Batching Works

Consider a linear layer:

y = xW

For a single input x ∈ ℝ^(1×d_in) (a row vector), you get output y ∈ ℝ^(1×d_out).

Now consider a batch of n inputs, stacked as rows:

    ┌ x₁ ┐
X = │ x₂ │  ∈ ℝ^(n×d_in)
    │ ⋮  │
    └ xₙ ┘

The batched computation:

Y = XW

gives you all n outputs in one matrix multiply. Same W, same operation—just more rows.

This only works because the operation is linear.

If f weren’t linear, you couldn’t factor through the batch. You’d have to compute f(x₁), f(x₂), … separately.

But for linear operations, batching is free—mathematically. You’re computing the same thing, just organized differently.

Why GPUs Love Linearity

Matrix multiplication is the most optimized operation in computing.

• NVIDIA Tensor Cores: Designed specifically for GEMM (General Matrix Multiply)

• Memory bandwidth: Amortized across the batch

• Parallelism: Thousands of multiply-adds happening simultaneously

When you increase batch size:

The weight matrix W is loaded once. Each additional sample in the batch is nearly free—you’re just doing more arithmetic while the data is already in fast memory.

Linearity turns “process n samples” into “one big matrix multiply.”

Gradient Accumulation

Here’s another consequence of linearity.

When you train on a batch, your loss is typically:

L = (1/n) Σᵢ Lᵢ

The gradient:

∇L = (1/n) Σᵢ ∇Lᵢ

Sum is linear. So:

• Compute gradients on samples 1-100, sum them

• Compute gradients on samples 101-200, sum them

• Add the partial sums

Same result as computing on all 200 at once.

This is gradient accumulation. When your batch doesn’t fit in memory, split it. Accumulate gradients across passes. Linearity guarantees correctness.

The same principle enables distributed training: compute gradients on different machines, sum them (all-reduce). Works because gradient aggregation is linear.

Why We Need Non-Linearity

If linearity is so great, why not make everything linear?

Because composition of linear functions is linear:

f(g(x)) = (xW_g)W_f = x(W_g W_f) = xW_combined

A 100-layer linear network equals a 1-layer linear network. No matter how deep you go, you can only learn linear functions.

Non-linearities create expressivity.

ReLU, GELU, softmax—these break linearity. They let deep networks approximate arbitrary functions.

The architecture of a neural network is:

Linear → Non-linear → Linear → Non-linear → ... → Linear

Linear operations: expensive, but batch-friendly, GPU-optimized. Non-linear operations: cheap (element-wise), parallel across the batch but no GEMM speedup.

This isn’t accidental. It’s engineered for hardware.

Where Linearity Breaks (And It Matters)

Batch Normalization

BatchNorm(x) = γ · (x - μ_B) / σ_B + β

The mean μ_B and standard deviation σ_B depend on which samples are in the batch.

Change the batch composition → change the normalization → change the output.

This is why:

• BatchNorm behaves differently in training vs. inference

• Small batches give noisy estimates

• BatchNorm can’t be cleanly gradient-accumulated

BatchNorm is not linear over the batch dimension.

Softmax in Attention

softmax(x)ᵢ = exp(xᵢ) / Σⱼ exp(xⱼ)

Every output depends on all inputs. You can’t compute softmax on parts and combine.

(Well, you can—that’s what we showed in the associativity article. But it requires the correction factor trick. It’s not trivially decomposable.)

Dropout

Stochastic. Different mask each time. Can’t be factored cleanly.

Backpropagation: Linearity of Differentiation

Here’s a deeper consequence.

Backpropagation relies on the chain rule:

∂L/∂x = ∂L/∂y · ∂y/∂x

But it also relies on differentiation being a linear operator:

∂/∂x(f + g) = ∂f/∂x + ∂g/∂x

∂/∂x(αf) = α · ∂f/∂x

Gradients add linearly. Scale linearly. This is why:

• Gradient of a sum = sum of gradients

• Gradient accumulation works

• Automatic differentiation is efficient

If differentiation weren’t linear, we couldn’t train neural networks.

The entire training paradigm—backprop, SGD, Adam—relies on gradients being linear in how they combine.

Practical Implications

Batch Size Tuning

Larger batches → better GPU utilization → faster per-sample processing.

But: larger batches can hurt generalization (sharper minima, less noise).

The trade-off is between:

• Hardware efficiency (wants large batches, because linearity makes them cheap)

• Optimization dynamics (sometimes wants smaller batches, for noise/regularization)

Gradient Checkpointing

To save memory, you can:

1. Discard intermediate activations during forward pass

2. Recompute them during backward pass

This works because the forward pass is deterministic—same input, same output. Recompute any segment, get identical activations, get identical gradients.

LoRA and Adapter Merging

Low-Rank Adaptation adds a small update:

W’ = W + BA

where B and A are low-rank matrices.

After training, you can merge the adapter back:

W_merged = W + BA

One matrix, no overhead at inference.

This works because matrix addition is linear. The adaptation is just a linear modification to the weights.

The Architecture of Efficiency

Modern neural networks are carefully designed around linearity:

Notice the trend: we use LayerNorm instead of BatchNorm in Transformers. Why? LayerNorm normalizes within each sample, not across the batch. It’s linear over the batch dimension.

Architecture choices reflect the desire to preserve linearity where it matters.

The Takeaway

Linearity is why batching works.

f(batch) = batch of f

For linear operations, processing a batch is just one big matrix multiply. GPUs are optimized for exactly this.

This single property enables:

• Batched inference: 1000 samples nearly as fast as 1

• Batched training: gradients over many samples at once

• Gradient accumulation: split batches, sum gradients

• Distributed training: sum gradients across machines

• Backpropagation itself: gradients combine linearly

Neural networks are towers of linear operations with strategic non-linearities. The linear parts enable efficiency. The non-linear parts enable expressivity.

Lose linearity carelessly, and you lose the ability to batch. That’s why BatchNorm is tricky. That’s why softmax needed FlashAttention.

The algebra isn’t abstract. It’s why training is tractable at all.

Next in this series: Domain Transformations—why logarithms prevent underflow, why Fourier transforms speed up convolutions, and the art of finding easier spaces.

See also: The One Property That Makes FlashAttention Possible — Associativity is the license to parallelize, chunk, and stream.

Further Reading

• Why Momentum Really Works — Optimization dynamics and batch size

• A Survey of Quantization Methods — Linear error accumulation in approximate computation

• LoRA: Low-Rank Adaptation — Exploiting linearity for efficient fine-tuning

• Batch Normalization — And why it complicates things

Next in this series: Domain Transformations—why logarithms prevent underflow, why Fourier transforms speed up convolutions, and the art of finding easier spaces.

📖 For the best reading experience with properly rendered equations, view this article on GitHub Pages.

Here’s a question that seems too simple to be interesting:

Why do Transformers need positional encodings?

The standard answer: “So the model knows where each token is in the sequence.”

But why doesn’t it know already? What is it about attention that loses position information?

The answer is a single word: commutativity.

The Property

Commutativity means order doesn’t matter:

a + b = b + a

Simple. Obvious. But the consequences run deep.

If an operation is commutative, permuting the inputs doesn’t change the output:

f(a, b, c) = f(c, a, b) = f(b, c, a)

The operation can’t tell what order things arrived in. It treats all orderings identically.

Attention Is Commutative

Look at self-attention:

Attention(Q, K, V) = softmax(QKᵀ / √d) × V

Given a sequence X = [x₁, x₂, …, xₙ], we compute:

• Q = XW_Q

• K = XW_K

• V = XW_V

Now permute the input: X’ = [x₃, x₁, x₂, …]

What happens? The output is permuted the same way.

Attention is permutation-equivariant. Shuffle the input, get a shuffled output. The operation itself doesn’t care about order.

This means without positional encodings:

"I love you"  →  [embed("I"), embed("love"), embed("you")]
"you love I"  →  [embed("you"), embed("love"), embed("I")]

These would produce the same representation (just permuted). The model literally cannot distinguish them.

Positional encodings exist because attention is commutative.

We add position information explicitly because the architecture won’t infer it.

The Design Choice

This commutativity is a feature, not a bug.

RNNs are explicitly non-commutative:

hₜ = f(hₜ₋₁, xₜ)

Each state depends on the previous state. Order is baked into the recurrence. You can’t permute the input without changing everything.

The cost: you can’t parallelize. Each step waits for the previous one.

Transformers are commutative (permutation-equivariant):

O = Attention(X)

Order is not baked in. You add it explicitly through positional encodings.

The benefit: you can parallelize. All positions are processed simultaneously.

Commutativity enables parallelization. This is why Transformers replaced RNNs for long sequences.

When You Want Commutativity

Sometimes order genuinely doesn’t matter. Then commutativity isn’t a limitation—it’s a requirement.

Point Clouds

A 3D scan gives you points {(x₁, y₁, z₁), (x₂, y₂, z₂), …}.

These points have no natural order. The first point scanned isn’t semantically “first.”

Your network must be permutation invariant:

f({p₁, p₂, p₃}) = f({p₃, p₁, p₂})

PointNet (2017) achieves this by design:

f(X) = g(maxᵢ h(xᵢ))

Max is commutative. Shuffle the points, get the same output.

Sets

A shopping cart. A user’s friends. Atoms in a molecule.

These are sets—order doesn’t exist.

DeepSets (2017) proved the fundamental theorem:

Any permutation-invariant function on sets can be written as: f(X) = ρ(Σ φ(x)) for all x ∈ X

Sum is commutative. The architecture is order-invariant by construction.

Graphs

In a graph neural network, you aggregate neighbor features:

hᵥ = UPDATE(hᵥ, AGG({hᵤ : u ∈ N(v)}))

The neighbors N(v) have no order. AGG must be commutative—sum, mean, or max.

When You Don’t Want Commutativity

Sometimes order is everything.

Language: “Dog bites man” ≠ “Man bites dog”

Time series: The sequence of stock prices matters

Music: Notes in order form melody; shuffled, they’re noise

Actions: The order of operations matters (usually)

For these, you need either:

1. Non-commutative operations (RNNs, state machines)

2. Explicit position encoding (Transformers)

The choice affects parallelization, inductive bias, and what the model can learn.

Gradient Aggregation: Why Training Works

Here’s a less obvious place commutativity matters.

When you train on a minibatch, you compute:

L = (1/N) Σ L(xᵢ, yᵢ)

∇L = (1/N) Σ ∇L(xᵢ, yᵢ)

The sum of gradients is commutative. This enables:

Minibatch training: Compute gradients for samples in any order, sum them.

Gradient accumulation: Split a large batch across multiple forward passes, sum the gradients.

Distributed training: Compute gradients on different machines, sum them (all-reduce).

If gradient aggregation weren’t commutative, distributed training would be impossible. The order of machines would matter.

Pooling: The Invariance/Information Trade-off

Global pooling—mean, max, sum—is commutative.

This gives you translation invariance:

[ cat in left of image  ] → pool → representation
[ cat in right of image ] → pool → same representation

The pooled representation doesn’t know where the cat was.

The trade-off: commutativity destroys positional information.

• Want invariance? Use commutative pooling.

• Want to preserve position? Don’t pool, or use position-aware alternatives.

This is why object detection uses feature pyramids instead of global pooling—you need to know where things are.

Designing with Commutativity

When building an architecture, ask:

Does order matter in my input?

Does order matter in my output?

If you’re generating sequences, you need autoregressive structure—fundamentally non-commutative.

Can I parallelize?

Commutative operations can be parallelized and distributed. Non-commutative ones often can’t.

The Floating-Point Footnote

One subtlety worth knowing:

In exact arithmetic, addition is both commutative (a + b = b + a) and associative ((a + b) + c = a + (b + c)).

In floating-point:

• Commutativity: preserved ✓

• Associativity: broken ✗

>>> (1e20 + (-1e20)) + 1
1.0
>>> 1e20 + ((-1e20) + 1)
0.0

Reordering is safe. Regrouping isn’t.

This means parallel reductions (which require regrouping) can give slightly different results on different runs.

This contributes to ML non-reproducibility, but research from Thinking Machines shows the primary culprit is subtler: GPU kernels change their reduction strategies based on batch size. When server load varies, batch sizes vary, kernel behavior varies—and you get different results even with identical inputs.

Either way, this is a numerical concern, not an architectural one. The mathematical design choice—commutative or not—remains valid.

The Takeaway

Commutativity is about order invariance.

Where order doesn’t matter—sets, point clouds, graphs—use commutative operations. You get parallelization and natural invariance.

Where order matters—sequences, time series—either use non-commutative operations (RNNs) or add position explicitly (Transformers).

Transformers need positional encodings because attention is commutative. The architecture processes all positions symmetrically. Order must be injected from outside.

This single property—commutativity—explains:

• Why Transformers parallelize and RNNs don’t

• Why PointNet works on point clouds

• Why GNNs use sum/mean/max aggregation

• Why global pooling loses spatial information

• Why distributed training is possible

The algebra isn’t abstract. It’s in every architecture you use.

See also: The One Property That Makes FlashAttention Possible — Associativity is the license to parallelize, chunk, and stream.

Further Reading

• Zaheer et al., “Deep Sets” (2017) — The foundational theorem on permutation-invariant functions

• Qi et al., “PointNet” (2017) — Processing point clouds with max pooling

• Vaswani et al., “Attention Is All You Need” (2017) — Transformers and positional encodings

• Bronstein et al., “Geometric Deep Learning” (2021) — Symmetry and invariance in neural networks

• Thinking Machines, “Defeating Nondeterminism in LLM Inference” (2025) — Why batch-invariance failure causes non-reproducibility

It’s in PyTorch, JAX, and every LLM serving stack. It delivers 2-4x speedups and cuts memory from O(n²) to O(n).

But ask practitioners why it works, and you get hand-wavy answers about “tiling” and “recomputation.”

Those are implementation details. Today I want to show you the actual why - the mathematical property that makes it all possible.

We’re not starting with attention though. We’re starting with something simpler: computing an average.

A Simple Problem: The Average

You have a billion numbers. You want their average.

The obvious approach:

One number at a time. One core. One long wait.

But here’s the thing: you don’t have to process them in order.

Split the data into chunks. Compute each chunk’s sum and count separately:

Then combine:

Same answer. But now you can process chunks in parallel.

Why This Works: Associativity

This works because addition is associative:

Grouping doesn’t change the result.

More precisely: the pair (sum, count) can be combined, and that combination is associative. This single property unlocks three capabilities:

One algebraic property, three capabilities. If your combination operation is associative, you can split, parallelize, and stream - automatically.

This might seem obvious for averages. But the same principle - find combinable state, verify associativity - applies to surprisingly complex operations.

Let’s try something harder.

Why Softmax Seems Different

Softmax is the heart of attention:

To compute any output, you need the sum over all elements.

If I show you half the values, you can’t produce final outputs. You’re missing half the denominator. Unlike our average, where partial sums combine cleanly, softmax seems to require everything upfront.

Can we find hidden associative structure anyway?

Before we answer, we need to address a practical problem.

The Overflow Problem

Here’s what breaks naive softmax:

When values are large, exp() overflows to infinity. And attention scores can be large.

The fix exploits a mathematical identity—softmax is translation-invariant:

for any constant c.

Subtracting the same value from every element doesn’t change the output. So we subtract the maximum:

Now every exponent is ≤ 0, so exp() never exceeds 1. No overflow.

This is why you see max everywhere in softmax code. It’s not algorithmic - it’s survival.

But now we have two things to track: the max and the sum. Let’s see if they combine.

The State for Softmax

For our average, the state was (sum, count).

For softmax, given values x₁, x₂, ..., xₙ, we need:

- m = max of all values

- s = sum of exp(xᵢ - m)

Can we combine two such pairs from separate chunks?

Why Simple Addition Fails

Say we have:

Chunk A: values [1, 2, 3]
         m_a = 3
         s_a = exp(1-3) + exp(2-3) + exp(3-3) ≈ 1.50
Chunk B: values [4, 5]
         m_b = 5
         s_b = exp(4-5) + exp(5-5) ≈ 1.37
The combined max is max(3, 5) = 5.

But we can’t just add the sums.

Chunk A’s sum was computed relative to max=3. Chunk B’s sum relative to max=5. They’re in different “units.”

Adding 1.50 + 1.37 = 2.87 would be wrong.

The Correction Factor

Here’s the key insight.

Chunk A computed

.

But relative to the true max of 5, it should be

These relate by:

So we correct chunk A’s sum:

Chunk B needs no correction (its max equals the combined max):

Combined:

Here’s what’s happening visually:

The correction factor exp(m_local - m_global) rescales the local sum to the global reference frame.

Let’s verify by computing directly on [1, 2, 3, 4, 5]:

Same answer. The combination works.

The Combine Rule

We’ve derived the combination for softmax state:

Is this associative?

Yes. The algebra works out:

The pair (max, scaled_sum) forms a monoid—same structure as (sum, count) for averages, just with a more complex combination.

---

The Pattern, Generalized

Same structure. Different complexity.

The softmax case requires correction because changing the max changes the meaning of the sum. But the fundamental pattern - combinable state, associative operation - is identical.

FlashAttention: Applying the Pattern

FlashAttention applies this to attention:

Instead of the full n×n matrix, it:

1. Processes tiles of K and V one at a time

2. Maintains running state: (max, sum, output)

3. Applies correction when a new tile reveals a larger max

4. Never materializes the full matrix

Every tile might reveal a larger max. The correction factor adjusts all previous work. Nothing needs recomputation—the associative structure guarantees consistency.

The Skill to Develop

FlashAttention’s genius isn’t tiling or memory tricks.

It’s recognizing that softmax has **hidden associative structure**.

This is a learnable skill. When facing a “global” computation:

1. What state do I need?

- Average: (sum, count)

- Softmax: (max, sum)

- Your problem: (?, ?)

2. Can partial states combine?

- Write the combine function

- Check: does order matter?

3. Is correction needed?

- Does new information change the meaning of old state?

- If so, what’s the adjustment factor?

4. Is it associative?

- If yes: parallelize, chunk, stream

- If no: fundamentally different algorithm needed

The Counter-Example

Not everything has this structure.

Median:

No way to derive 3.5 from 2 and 5. No combinable state exists.

That’s why median requires sorting or selection—fundamentally different algorithms.

Knowing what can’t be expressed this way is as valuable as knowing what can.

The Takeaway

FlashAttention works because of a mathematical property.

Softmax has associative structure hidden beneath its global-looking surface. The (max, scaled_sum) pair combines across chunks. The combination requires correction, but it’s still associative.

Associativity is the license to parallelize, chunk, and stream.

The property came first. The algorithm followed.

This is the difference between collecting tricks and understanding foundations. Tricks solve one problem. Properties solve a category of problems.

When you face a seemingly global computation, don’t reach for a trick.

Reach for the algebra.

Next week: Why commutativity enables parallelization - and when floating-point arithmetic breaks it.

This series will cover the complete algebraic toolkit for ML performance: commutativity, distributivity, linearity, and the domain transformations that make numerical computing stable.

PyTorch is a popular open-source machine learning library known for its flexibility and dynamic computation graph. However, when it comes to optimizing certain mathematical operations, it falls short. One example of this is its approach to matrix multiplication, which, in some cases, lacks basic optimizations. In this article, we'll explore PyTorch's shortcomings when it comes to optimizing matrix multiplication and discuss a specific scenario where it doesn't use the most efficient approach.

Matrix Multiplication in PyTorch

Matrix multiplication is a fundamental operation in various machine learning and scientific computing tasks. It forms the backbone of many deep learning models, including neural networks. To perform matrix multiplication, PyTorch offers several methods, including `torch.mm()`, `torch.matmul()`, and the `@` operator. While these methods are versatile and easy to use, they may not always perform optimally.

The Problem with Basic Matrix Multiplication Optimization

Matrix multiplication can be a computationally intensive task, especially when dealing with large matrices. In a naive approach to matrix multiplication, to compute the product of two matrices, A and B, you would typically iterate over the rows and columns of the matrices and perform element-wise multiplications and additions. This approach, also known as the standard matrix multiplication algorithm, has a time complexity of O(n^3), where n is the size of the matrices.

To improve efficiency, more advanced algorithms such as Strassen's algorithm and the Coppersmith–Winograd algorithm have been developed, which reduce the number of elementary multiplications needed to compute the result. In the context of deep learning, PyTorch utilizes highly optimized and efficient matrix multiplication libraries like Intel MKL or NVIDIA cuBLAS when available, which can significantly speed up these operations.

However, when it comes to deciding which matrices to optimize, PyTorch can easily miss basic optimization opportunities.

An Illustrative Example: Computing the Fourth Power of a Matrix

Let's look at a specific example where PyTorch fails to apply a basic optimization. Suppose we want to compute the fourth power of a matrix A. One straightforward approach is to multiply the matrix by itself four times:

A^4 = A * A * A * A

In this case, PyTorch's default matrix multiplication method will perform a matrix multiplication three times, resulting in four multiplications, even though it can be done with just two multiplications. This is because PyTorch does not recognize the inherent mathematical symmetry and uses a naively repeated multiplication approach.

For a random 1000x1000 matrix it takes ~92ms to perform matmul4

Attentive reader may suggest using pytorch.compile

but, unfortunately, this doesn’t make much difference

Optimizing the Computation of A^4

To compute the fourth power of a matrix more efficiently, one can exploit the properties of matrix exponentiation. Using exponentiation by squaring, you can compute A^4 as follows:

A^4 = (A^2)^2

This method reduces the number of matrix multiplications, and in this case, only two multiplications are required: A^2 and (A^2)^2. Using this optimized approach can significantly reduce the computational burden when dealing with large matrices.

The code to accomplish this is trivial

and results in a desired speedup

The same can be achieved by using

torch.matrix_power(mtx, 4)

that delivers similar performance

Note that this optimization relies on associativity and since floating point multiplications are non-associative, the result of optimized version is generally not equal to non-optimized one. This doesn’t mean it’s not correct and the difference is certainly much smaller than the error introduced by quantization and other hacks that are often employed to speed up learning or inference. On the bright side performance gap between optimized and non-optimized versions grows with matrix dimensions and the power.

Conclusion

While PyTorch is a powerful and widely used library for deep learning and scientific computing, it's not immune to optimization issues. In the case of matrix multiplication, PyTorch may not always apply the most efficient high-level algorithms. To achieve the best performance in such cases, users must be aware of these limitations and apply their own optimizations when necessary. Understanding the underlying algorithms and mathematical properties is crucial to harnessing PyTorch's full potential and ensuring efficient computation in complex machine learning tasks.

You can play with above examples using colab playground.

Subscribe now

Code snippet above from glibc library does exactly what we’d expect - looks for the first pair of characters that are different and returns their difference. Obviously this code assumes that passed strings are null-terminated and its behavior is undefined otherwise. Oftentimes, at least one of the strings is known at compile-time, so we can bound the number of traversed charecters. strncmp accepts a third argument that indicates the maximum number of characters that are used for comparison. It’s easy to extend strcmp implementation to take into account the maximum number, but interestingly glibc’s implementation of strncmp looks a little different

It does loop unrolling to handle comparisons in blocks of 4. It seems like a reasonable thing to do to reduce the number of n > 0 checks but since it does strictly more work than strcmp it seems like it should be slower, right?

Let’s find out.

And the results are

2.2X faster! Don’t forget that the numbers can be different depending on the inputs, compilers and standard library implementations, so don’t take my word for it and run your own benchmarks. My main point point is that since strncmp is a little more safe than strcmp performance shouldn’t be a reason you prefer strcmp over strncmp.

The importance of reservoir sampling lies in its ability to provide a truly random and unbiased sample, regardless of the dataset's size. This characteristic is particularly valuable when dealing with data streams, massive databases, or situations where you need to maintain the statistical integrity of your sample. Reservoir sampling ensures that every item in the dataset has an equal chance of being included in the sample, making it an indispensable tool for designing efficient and accurate algorithms in computer science and data analysis. Whether you're working with big data or designing randomized algorithms, reservoir sampling is a key technique to ensure the robustness and reliability of your results.

In fact it’s so important that it used to be a popular coding interview question at largest tech companies. Fortunately its most popular implementation is fairly trivial

It’s easy to notice that random number has to be generated for every element from the stream after reservoir is filled. This means that if n » k random generator calls are likely to impact overall performance. What if instead of throwing a dice for every element from the stream to see if it should end up in the reservoir, we could somehow come up with a strategy of skipping certain number of source elements before accepting one into the reservoir. Turns out there is a way. Theory behind such algorithms is tricky but well covered in other articles, so here I’ll only consider AlgorithmL because of its simplicity

It’s certainly more involved but not much more code. Is this extra complexity justified though? Let’s see

and the results look very promising with ~20X speedup.

Better algorithms can bring impressive wins, so it never hurts to follow the latest research papers.

It’s not a secret that branching is not particularly performance friendly - it complicates life to optimizer and creates more work for branch predictor. This code is translated into very much expected

with cmp and branch used for each if.

After staring at this code for a few seconds I’ve noticed that it follows a fairly predictable pattern - each chat32_t goes through a ladder with each step adding +1 to final utf8 length. It’s easier to see this in code

Note that this implementation is branchless - we just add 0 or 1 depending on whether current character exceeds certain threshold. This straightforward implementation is much easier for compiler to grok, so it figures that it’s possible to vectorize it

Sure, it’s more verbose but we are now using the power of SIMD, right?

Let’s see

And the results are

What a bummer! For regular ASCII string we take a significant performance hit. For mixed unicode strings results are not as bad

but still not something we were hoping for.

My attempt landed in ada-url as part of https://github.com/ada-url/idna/pull/33 but the lesson here is - don’t assume SIMD is always magically better and trust only benchmarks using representative inputs.

You can look for the answer “why” using tools like uiCA or reading hardware manuals but at the end of the day what matters is measured result.

When it comes to SIMD, we usually think about SSE*, AVX and Neon, but vectorization has been used for years before any of them appeared. Let’s take a look at a common parsing subroutine to check if string has a hex prefix ‘0x’. The first thing that comes to mind is something like

which, by using a trick for fast lowercase transformation, can be reduced to

Clang generates the same assembly for both versions

It may seem like there is nothing else to be done here. But we are dealing with just 2 single byte characters here, so why do we have to process them separately? What if we convert first 2 chars into a 16bit integer so that they could be processed as a single unit? And that’s exactly what ada library for URL parsing does

Since it’s a generic library, it has to take into account endianness so that the second character is converted to lowercase using `| 0x20` trick we’ve used above. Also, it’s a UB to reinterpret_cast string bytes into uint16_t, so std::memcpy is has to be used until C++20’s std::bit_cast can be used.

Surely this implementation would result in a lot of assembly, right?

Turns out that optimizing compilers like clang easily get rid of memcpys and in the end everything boils down to and, cmp and sete.

In this article we had to process just 2 chars, but even without going all the way to something like AVX512, we can easily process 8 chars using uint64_t available on most platforms.

So when we write functions, we establish relationship between its parameters and its return values. In mathematics xs and ys are just symbols and functions do not have cost. In software it’s a different story - values take memory and function execution uses CPU cycles, so performance work is a search for cheaper way to establish relations.

That’s why we can use squares of distances between points instead of distances to avoid computing square roots. But today we’re going to use a different example. Rust is known for its developer friendly error messages, so we are going to look at

used for finding best suggestion candidates.

Note, that sort_by_words returns an owned String that is a formed by joining sorted parts of the original name split by underscore. Thanks to Rust’s lifetimes split returns slices of the name without new allocations, but, unfortunately, .join(“_”) cannot return a slice into name and as such has to create a new string.

But wait a second - do we actually need this joined string? Think about what kind of relation sort_by_words(name1) == sort_by_words(name2) establishes between name1 and name2. Is it any different than if we were to return split_words vector intead of joined string? It’s fairly obvious that for all strings name1 and name2, if vec![n11,n12,..] == vec![n21,n22,…] then “${n11}_${n12}_…}” == “${n21}_${n22}_…}” and the other way around. This means that we can rewrite sort_by_words as follows

and get the same result without unnecessary string allocations.

It’s time to see how much difference does it make using

and the results on M1 MacBook Air are

     Running benches/sort_by_words_benchmark.rs (target/release/deps/sort_by_words_benchmark-5adee33fa37ce7e2)
Benchmarking multiply add/original: Collecting 100 samples in estimated 5.0006 s (27M iteratio
multiply add/original   time:   [172.85 ns 177.30 ns 182.83 ns]
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe
Benchmarking multiply add/proposed: Collecting 100 samples in estimated 5.0000 s (61M iteratio
multiply add/proposed   time:   [81.106 ns 81.605 ns 82.261 ns]
Found 12 outliers among 100 measurements (12.00%)
  5 (5.00%) high mild
  7 (7.00%) high severe

In summary we’re getting a 2X+ faster code without any downsides. That’s the power of relation modeling in action. PR has already landed in Rust and may already be processing your compiler errors.

Attentive readers must have noticed the .unwrap() call on the result of from_utf8. It’s required because from_utf8 returns Err if the slice is not UTF-8 with a description as to why the provided slice is not UTF-8. It’s pretty obvious that in order to decide if error should be returned, from_utf8 has to validate provided slice, which is not free.

Fair enough, but is it possible for the slice to ever contain non UTF-8 characters? All characters come from BASE_64 array and all of its characters are valid UTF-8 characters. In such case, why not skip the validation?

Of course, we leave a safety comment explaining why this unsafe block is actually safe and use the following benchmark to see if it makes any difference

The results on M1 macbook air:

Running benches/base_n_benchmark.rs (target/release/deps/base_n_benchmark-825fe5895b5c2693)
push_str/old            time:   [14.670 µs 14.852 µs 15.074 µs]
Found 11 outliers among 100 measurements (11.00%)
  4 (4.00%) high mild
  7 (7.00%) high severe
push_str/new            time:   [12.573 µs 12.674 µs 12.801 µs]
Found 11 outliers among 100 measurements (11.00%)
  7 (7.00%) high mild
  4 (4.00%) high severe

Not bad for a single liner? Compilers have a difficult job of optimizing code without affecting correctness. Sometimes we have knowledge that is outside of compiler’s jurisdiction and can help it by taking responsibility on maintaining some of the invariants on our own. When making a decision to do this, don’t forget to consider the cost of maintenance - when assumptions or code change, someone has to go through manually maintained invariants to make sure they are still up-to-date. In this particular case, the change is fairly localized and shouldn’t be hard to maintain, but you can see that reviewers are not rushing to accept it.

The implementation is clean and easy to understand. At the same time, every time I see reverse call, I start to wonder if it’s really necessary. Let’s see, on each iteration we chop off the last digit of the resulting number and we place it at the `index`th position from left to right. But we already know the size of the buffer, so why don’t we just place this digit into its final position right away?

This way we no longer need the reverse call. We shouldn’t expect huge wins here, since the length of the resulting string is usually fairly small to matter much, but according to

this small change even managed to result in a modest win on M1 macbook air:

Running benches/base_n_benchmark.rs (target/release/deps/base_n_benchmark-825fe5895b5c2693) push_str/old time: [14.180 µs 14.313 µs 14.462 µs] Performance has improved. Found 5 outliers among 100 measurements (5.00%) 4 (4.00%) high mild 1 (1.00%) high severe push_str/new time: [13.741 µs 13.839 µs 13.973 µs] Performance has improved. Found 8 outliers among 100 measurements (8.00%) 3 (3.00%) high mild 5 (5.00%) high severe

There is also a chance that at this point you’re using rustc with this change.

Attentive reader would notice that this function is essentially std::equal_range from C++ but with much more involved implementation. Even with usual std lib uglifiers, C++ LLVM implementation is much easier to understand and reason about:

The reason behind its simplicity is the fact that it uses std::lower_bound and std::upper_bound to locate range start and end.

Unfortunately, Rust doesn’t have a lower/upper bound functions, but we can use partition_point to simulate both. Using this function we can rewrite binary_search_slice as follows:

I’ve added comments with invariants so that readers can easily convince themselves in correctness of proposed implementation and it’s still significantly shorter than original implementation. In fact, it’s possible to make it even shorter by skipping the if on line 10 and unconditionally returning &data[start..end], since in case there are no elements with matching key, start and end would be equal making start..end range empty. The reason I decided not to go with this implementation is because it would increase the number of comparisons in the worst case.

Since there is a popular myth that performant code is more complex, some readers would start wondering about performance implications of such change. Let’s find out using the following benchmark

And the results on my M1 macbook air are

suggesting ~20% runtime improvement. Not bad for a code cleanup?

This is yet another reminder to follow Alexander Stepanov’s algorithm design principles and leverage algorithm composition.

P.S.: these suggestions are part of https://github.com/rust-lang/rust/pull/114231

Bentley's implementation of binary search was correct for most cases, but it contained a subtle bug that could cause it to fail for large arrays. The bug was in the line of code that calculated the midpoint of the array. Bentley's code used the following expression to calculate the midpoint:

int mid = (low + high) / 2; 

This expression works correctly for most cases, but it can overflow for large values of the variables low and high. In fact, helpful compilers can point out such bugs in obvious cases like the one below

warning: overflow in expression; result is -4 with type 'int' [-Winteger-overflow]
return (std::numeric_limits<int>::max() - 2 + std::numeric_limits<int>::max()) / 2;

This means that if low is std::numeric_limits<int>::max() - 2 and high is std::numeric_limits<int>::max(), then the expression (low + high) / 2 will overflow and produce the value -4 and result in undefined behavior.