Notes on Linear Attention: From Linear Attention to KDA

This note follows one central question:

Can we reformulate attention so that, instead of looking back at every previous token for each query, past tokens continually update a fixed-size associative memory that the current query simply reads from?

Linear Attention provides the most basic version of this fixed-size memory. DeltaNet allows the memory to correct and overwrite old associations. Gated DeltaNet adds global forgetting, and Kimi Delta Attention (KDA) refines that mechanism further by controlling retention independently along each key-feature dimension.

Throughout the note, I consider a single causal attention head and use the row-vector convention that most directly matches the PyTorch reference code:

$q_t,k_t\in\mathbb R^{1\times d_k}$ and $v_t,y_t\in\mathbb R^{1\times d_v}$ ;
the memory matrix is $S_t\in\mathbb R^{d_k\times d_v}$ , with readout $q_tS_t$ ;
the multi-head case applies the same derivation independently to each head;
output projection, RMSNorm, short convolution, and the output gate are omitted so that we can focus on the core structure.

A one-dimensional PyTorch tensor such as q_t with shape [..., d_k] is not intrinsically a row or column vector: transposing it alone does nothing. The orientation comes from how dimensions are contracted. FLA’s reference KDA implementation stores the state as [..., d_k, d_v] and uses contractions equivalent to

prediction = torch.einsum("...k,...kv->...v", k_t, state)
state = state + torch.einsum("...k,...v->...kv", beta_t[..., None] * k_t, v_t - prediction)
y_t = torch.einsum("...k,...kv->...v", q_t, state)

These operations are exactly $k_tS_{t-1}$ , $\beta_tk_t^\top(v_t-k_tS_{t-1})$ , and $q_tS_t$ in row-vector notation. Some papers instead use column vectors and a $d_v\times d_k$ state. That convention is the transpose of every state equation below; both are equivalent when used consistently.

1. Starting from softmax attention

For the $t$ -th token, causal softmax attention is

y_t = \sum_{i=1}^{t} \frac{\exp(q_t k_i^\top/\sqrt{d_k})} {\sum_{j=1}^{t}\exp(q_t k_j^\top/\sqrt{d_k})} v_i.

It performs two operations:

compare $q_t$ with every previous $k_i$ to decide where to attend;
use those scores to compute a weighted sum of the corresponding $v_i$ .

Its main advantage is that every query can directly access every historical key–value pair. The cost is that

a naively materialized training-time attention matrix contains $O(L^2)$ entries;
autoregressive decoding requires a KV cache that grows with sequence length;
at step $t$ , the query must be compared with all $t$ previous keys.

FlashAttention substantially improves IO efficiency and avoids materializing large intermediate tensors, but it does not change the underlying mathematical structure: every query still interacts with all historical keys.

2. Why separate Q from K, but combine K and V?

This is the most important step in understanding Linear Attention.

The temporal roles of Q, K, and V are asymmetric. A pair $(k_i,v_i)$ must remain available to the current query and every future query, whereas $q_t$ is used only to produce the output at the current position.

2.1 Softmax fixes the order of computation

Ignoring normalization for a moment, attention can be written in matrix form as

Y=(QK^\top)V.

The usual evaluation order first computes

A=QK^\top\in\mathbb R^{L\times L},

and then computes $AV$ . This corresponds to comparing every query with every key. The sequence dimension $L$ appears twice, which leads to quadratic complexity.

If the similarity function is an ordinary inner product, however, matrix multiplication is associative:

(QK^\top)V=Q(K^\top V).

Under the row-vector convention used here, the state on the right-hand side is

S_t=\sum_{i\le t}k_i^\top v_i, \qquad y_t=q_tS_t.

This is what it means to separate Q from K while combining K and V.

2.2 Intuition: a query is a one-time request; key–value pairs form memory

A historical key and value together describe an association:

k_i\longmapsto v_i.

The outer product $k_i^\top v_i$ can be interpreted as writing this association into a linear map. Summing all historical associations gives $S_t$ . The current query does not need to participate in storing history; it only serves as an address at read time:

q_t\longmapsto q_tS_t.

Therefore:

K and V are combined because both belong to the history and together form an address–content pair that should persist;
Q is separated from K because Q belongs to the current read operation, so there is no need to create and store an intermediate result between it and every historical key;
more fundamentally, associativity lets us replace a token-by-token similarity matrix with a feature-to-feature memory state.

2.3 Why can we not simply reorder softmax attention?

Because

$\operatorname{softmax}(QK^\top)V$ ≠ $Q\bigl(\operatorname{softmax}(K^\top)V\bigr)$

Softmax is a nonlinear operation applied, for each query, across all key scores. That nonlinearity destroys the associativity required by the reordering. Linear Attention does not magically rearrange softmax; it replaces the softmax kernel with one that can be factorized.

3. Kernelization and the derivation of Linear Attention

Write the softmax similarity as a kernel:

\kappa(q,k)=\exp(qk^\top/\sqrt{d_k}).

Suppose we choose a feature map $\phi:\mathbb R^{1\times d_k}\to\mathbb R^{1\times d_\phi}$ such that

\kappa(q,k)\approx \phi(q)\phi(k)^\top.

Then

y_t = \frac{ \sum_{i\le t}\bigl(\phi(q_t)\phi(k_i)^\top\bigr)v_i }{ \sum_{i\le t}\phi(q_t)\phi(k_i)^\top }.

We can now group together the terms that depend only on the past:

S_t=\sum_{i\le t}\phi(k_i)^\top v_i, \qquad z_t=\sum_{i\le t}\phi(k_i),

which gives

y_t = \frac{\phi(q_t)S_t}{\phi(q_t)z_t^\top}.

The recurrent form is

\begin{aligned} S_t &= S_{t-1}+\phi(k_t)^\top v_t,\\ z_t &= z_{t-1}+\phi(k_t),\\ y_t &= \frac{\phi(q_t)S_t}{\phi(q_t)z_t^\top+\varepsilon}. \end{aligned}

This reveals the RNN, or fast-weight-memory, interpretation of Linear Attention: $S_t$ and $z_t$ are fixed-size recurrent states.

Later DeltaNet-style models commonly use normalized queries and keys and omit the explicit denominator state $z_t$ . When moving into DeltaNet below, I therefore write $k_t$ and $q_t$ directly instead of $\phi(k_t)$ and $\phi(q_t)$ . This does not mean that every form of Linear Attention lacks a denominator; it reflects a different parameterization.

3.1 In what sense is the complexity linear?

Assuming $d_k$ and $d_v$ are fixed:

updating $S_t$ costs $O(d_vd_k)$ per token;
reading $q_tS_t$ costs $O(d_vd_k)$ per token;
a sequence of length $L$ costs $O(Ld_vd_k)$ in total;
the decoding state occupies $O(d_vd_k)$ memory and does not grow with context length.

“Linear” refers to sequence length $L$ , not to the head dimensions. A fixed-size state is not free either: when the state is large, decoding can become limited by the bandwidth required to read and write it.

3.2 What do we lose?

Softmax attention preserves every key and value, allowing a future query to decide exactly which token to retrieve. Linear Attention compresses an arbitrarily long history into a fixed-size matrix, creating an unavoidable information bottleneck:

writes interfere when keys are not orthogonal;
a finite-dimensional state cannot losslessly preserve infinitely many associations;
the simplest additive update only accumulates information and cannot actively revise an old association.

The final limitation leads directly to DeltaNet.

4. The problem with additive memory: it can write, but not revise

Consider the simplest state update:

S_t=S_{t-1}+k_t^\top v_t.

It is important to distinguish two different operations.

Normal attention readout uses the current query:
$y_t=q_tS_t.$
If we keep the feature-map notation from Section 3, this becomes $y_t=\phi(q_t)S_t$ , optionally with denominator normalization.
Inspecting or updating a particular key–value association asks the state, “What is currently stored at address $k$ ?” and therefore uses
$\hat v(k)=kS.$
With an explicit feature map, the more precise expression is $\hat v(k)=\phi(k)S$ . DeltaNet instead feeds an L2-normalized $k$ directly into the state, so I will use the shorter notation $kS$ from here on.

Thus, evaluating $kS$ below does not mean that we replace the query with the key during normal attention. It is a read-before-write check: to update an association, we first need to know what the old memory already predicts at that address.

Suppose the same key $k$ is first bound to $v_1$ and later should be rebound to $v_2$ . After two additive writes,

kS = k(k^\top v_1+k^\top v_2) = \lVert k\rVert^2(v_1+v_2)

if we temporarily ignore interference from other keys. The memory returns the sum of the old and new values rather than replacing the old value.

A better update should not blindly add $v_t$ . It should first ask:

What does the current memory already predict at $k_t$ , and how far is that prediction from the new target $v_t$ ?

Writing only this error gives us the delta rule.

5. DeltaNet: treating the state as an online linear model

5.1 Derivation from an online regression objective

Interpret $S$ as an online linear model that receives a key and predicts a value:

\hat v_t=k_tS_{t-1}.

For the current sample, define the squared error

\mathcal L_t(S) = \frac12\lVert k_tS-v_t\rVert_2^2.

Its gradient with respect to $S$ is

\nabla_S\mathcal L_t =k_t^\top(k_tS-v_t).

Taking one gradient-descent step with learning rate $\beta_t\in[0,1]$ gives

\boxed{ S_t = S_{t-1}+\beta_tk_t^\top(v_t-k_tS_{t-1}) }

while the normal readout remains

y_t=q_tS_t.

This is the core DeltaNet update. The model usually generates $\beta_t$ dynamically from the current input; it acts as a write gate that controls how strongly or confidently the current association should be updated.

5.2 The erase-and-write view

Expanding the update gives

\begin{aligned} S_t &=S_{t-1} -\underbrace{\beta_tk_t^\top(k_tS_{t-1})}_{\text{erase the old prediction along }k_t} +\underbrace{\beta_tk_t^\top v_t}_{\text{write the new target}}\\ &=(I-\beta_tk_t^\top k_t)S_{t-1}+\beta_tk_t^\top v_t. \end{aligned}

DeltaNet therefore does not indiscriminately clear the whole memory. It only corrects the map along the current key direction.

5.3 Why L2-normalize the key?

Let $\lVert k_t\rVert_2=1$ . Reading immediately after the update with the same key gives

\begin{aligned} k_tS_t &=k_tS_{t-1} +\beta_t(k_tk_t^\top)(v_t-k_tS_{t-1})\\ &=(1-\beta_t)k_tS_{t-1}+\beta_tv_t. \end{aligned}

Therefore:

if $\beta_t=1$ , then $k_tS_t=v_t$ : the association is exactly overwritten along that direction;
if $0<\beta_t<1$ , the result interpolates between the old prediction and the new target;
without normalization, the effective step size is also multiplied by $\lVert k_t\rVert^2$ , making write strength harder to control.

The update can still affect other keys that are similar to $k_t$ . This is cross-talk in a finite-dimensional associative memory, not something the delta rule can eliminate completely.

5.4 Why the state transition is harder to parallelize

Define

H_t=I-\beta_tk_t^\top k_t.

Then

S_t=H_tS_{t-1}+\beta_tk_t^\top v_t.

$H_t$ is an identity-minus-rank-one, generalized Householder-style transformation. Across multiple steps, we obtain an ordered product

H_tH_{t-1}\cdots H_1.

This is harder to parallelize across the sequence than the prefix sum used by additive Linear Attention. An important contribution of the 2024 DeltaNet work was to construct a hardware-friendly chunkwise algorithm using compact WY and Householder representations. Training can then proceed in parallel within chunks, while decoding still uses the simple recurrent update.

Two ideas should remain separate:

the model rule: the delta update improves overwrite behavior and associative recall;
the implementation: the chunkwise parallel form makes that rule practical to train on GPUs.

6. Why add a gate? DeltaNet does not forget proactively

The delta rule corrects the current key direction. But if an old piece of information is never followed by a similar key, it may remain in the state indefinitely. For language modeling, we often want the model to decide how long the memory as a whole should persist.

Gated DeltaNet (GDN) introduces a scalar retention gate $\alpha_t\in(0,1]$ . A clear way to write the update is to decay the old state first:

\bar S_{t-1}=\alpha_tS_{t-1},

and then apply the delta rule to the decayed memory:

\boxed{ S_t = \bar S_{t-1} +\beta_tk_t^\top(v_t-k_t\bar S_{t-1}) }

or equivalently,

S_t = \alpha_t(I-\beta_tk_t^\top k_t)S_{t-1} +\beta_tk_t^\top v_t.

The two gates have different roles:

$\alpha_t$ is the retention, or forget, gate: it controls how much of the old state survives;
$\beta_t$ is the update, or write, gate: it controls how strongly the current association is corrected.

But $\alpha_t$ is a scalar. Whenever the model forgets, every key-feature direction decays by the same amount. Different features in a finite state may represent different kinds of information or operate on different time scales, so this all-or-nothing decay limits expressivity.

7. Kimi Delta Attention: from scalar to fine-grained retention

KDA makes one focused change: it replaces GDN’s scalar retention $\alpha_t$ with a vector

\boldsymbol\alpha_t\in(0,1]^{d_k}, \qquad D_t=\operatorname{Diag}(\boldsymbol\alpha_t).

The gate acts along the key-feature axis of the state. First define the decayed memory

\bar S_{t-1}=D_tS_{t-1},

then apply the familiar delta correction:

\boxed{ S_t = \bar S_{t-1} +\beta_tk_t^\top(v_t-k_t\bar S_{t-1}) }

Expanding this gives KDA’s recurrent form under our $d_k\times d_v$ state convention:

\boxed{ S_t = (I-\beta_tk_t^\top k_t)D_tS_{t-1} +\beta_tk_t^\top v_t, \qquad y_t=q_tS_t. }

7.1 Why is fine-grained retention more expressive?

GDN gives the entire key-feature space a single retention time scale:

S_{t-1}\mapsto \alpha_tS_{t-1}.

KDA lets different directions retain information for different lengths of time:

S_{t-1}\mapsto \operatorname{Diag}(\boldsymbol\alpha_t)S_{t-1}.

Some dimensions can decay rapidly to track local syntax, while others stay close to one and preserve long-range semantics. Because the gate is generated dynamically from the input, the time scale of a given feature can also change from token to token.

This mechanism also provides an implicit relative positional signal. Temporarily ignore the rank-one delta correction and consider only diagonal decay. A write introduced at position $i$ is multiplied, by the time it reaches position $t$ , by

\prod_{j=i+1}^{t}D_j.

The resulting memory depends both on distance and on which intervening tokens were encountered, and each feature can follow a different decay trajectory. In Kimi Linear, this allows the interleaved MLA layers to use NoPE while recurrent KDA supplies positional awareness. That is a property of the complete hybrid architecture, however; it should not be generalized into the claim that KDA never needs positional encoding in every possible setting.

7.2 KDA still follows the delta rule

Let

\bar S_{t-1}=D_tS_{t-1}.

The KDA update remains

S_t=\bar S_{t-1}+\beta_tk_t^\top(v_t-k_t\bar S_{t-1}).

Every step still performs the same sequence:

obtain the decayed memory used at the current step;
read its old prediction at $k_t$ ;
compute the residual between the target $v_t$ and that prediction;
write only the residual back along the $k_t$ direction.

If $\lVert k_t\rVert=1$ and $\beta_t=1$ , then $k_tS_t=v_t$ still holds. Fine-grained forgetting does not break the overwrite property of the delta rule.

8. Why is KDA a special DPLR recurrence?

KDA’s state-transition matrix is

P_t=(I-\beta_tk_t^\top k_t)D_t.

Expanding it gives

P_t =D_t-\beta_tk_t^\top(k_tD_t).

This is a Diagonal Plus Low Rank matrix—more specifically, diagonal minus rank one:

P_t=D_t-b_ta_t^\top, \qquad b_t=\beta_tk_t^\top, \quad a_t=D_tk_t^\top.

The recurrence can therefore be written as

S_t=P_tS_{t-1}+\beta_tk_t^\top v_t.

A general DPLR recurrence may obtain its two low-rank vectors from independent projections. That is highly expressive, but its chunkwise expansion requires more pairwise matrix products, additional numerical-stability machinery, and more computation. KDA ties both sides of the low-rank correction to the same normalized key, with one side modulated by the diagonal gate. This constrained structure preserves a clear delta-rule interpretation and enables a more efficient specialized chunkwise kernel.

If vectors are instead written as columns, define $\widetilde k_t=k_t^\top$ , $\widetilde v_t=v_t^\top$ , and $\widetilde S_t=S_t^\top\in\mathbb R^{d_v\times d_k}$ . Transposing the row-vector recurrence gives

\widetilde S_t =\widetilde S_{t-1}D_t(I-\beta_t\widetilde k_t\widetilde k_t^\top) +\beta_t\widetilde v_t\widetilde k_t^\top.

This is the column-vector convention previously used in this note, not a different algorithm. When comparing a paper with an implementation, first check the state orientation and only then compare multiplication order.

9. Intuition for chunkwise computation

The recurrent form is ideal for autoregressive decoding: each incoming token updates a fixed-size state once. Training, however, should process many tokens in parallel rather than loop through the sequence in Python.

Split the sequence into chunks of size $C$ . Within a chunk, repeatedly substitute

S_t=P_tS_{t-1}+u_t, \qquad u_t=\beta_tk_t^\top v_t,

which yields

S_t = P_tP_{t-1}\cdots P_1S_0 +\sum_{i=1}^{t}P_tP_{t-1}\cdots P_{i+1}u_i,

where the transition product is the identity when $i=t$ .

This separates the work inside a chunk into three pieces:

propagating the state at the start of the chunk through a sequence of transitions;
propagating each within-chunk write to later positions;
retaining only a much shorter recurrence across chunk boundaries.

DeltaNet exploits the compact representation of rank-one Householder-style transitions. KDA exploits its special diagonal-minus-rank-one DPLR structure. Both aim to turn token-by-token recurrence into the large matrix multiplications GPUs handle well, without explicitly constructing every $d_k\times d_k$ transition product.

The exact kernel derivation involves WY representations, triangular systems within each chunk, and numerical-stability techniques. These are implementation-level optimizations. To understand the model itself, it is usually clearer to internalize the recurrent form before working backward from kernel code.

10. Three generations in one equation

Write all of the methods as

S_t=A_tS_{t-1}+B_t, \qquad y_t=q_tS_t.

Method	$A_t$	$B_t$	Core capability
Additive Linear Attention	$I$	$k_t^\top v_t$	Fixed-size additive associative memory
DeltaNet	$I-\beta_tk_t^\top k_t$	$\beta_tk_t^\top v_t$	Erases and rewrites the prediction along the current key direction
Gated DeltaNet	$\alpha_t(I-\beta_tk_t^\top k_t)$	$\beta_tk_t^\top v_t$	Adds global forgetting to the delta update
KDA	$(I-\beta_tk_t^\top k_t)D_t$	$\beta_tk_t^\top v_t$	Adds fine-grained retention along key features

Here, $D_t=\operatorname{Diag}(\boldsymbol\alpha_t)$ . The progression worth remembering is

\underbrace{I}_{\text{accumulate only}} \quad\longrightarrow\quad \underbrace{I-\beta k^\top k}_{\text{correct}} \quad\longrightarrow\quad \underbrace{\alpha(I-\beta k^\top k)}_{\text{forget globally}} \quad\longrightarrow\quad \underbrace{(I-\beta k^\top k)D}_{\text{forget selectively}}.

11. A two-dimensional example

Let $d_k=2$ and choose two orthogonal keys:

k_a=\begin{bmatrix}1&0\end{bmatrix}, \qquad k_b=\begin{bmatrix}0&1\end{bmatrix}.

The two rows of the state contain the values associated with $k_a$ and $k_b$ .

Additive update

If we first write $v_1$ at $k_a$ and then write $v_2$ at the same address, the first row becomes $v_1+v_2$ . There is no replacement semantics.

Delta update

If the second write uses $\beta=1$ , the update first subtracts the current contents of the first row and then writes $v_2$ . The first row becomes exactly $v_2$ , while the second row is unaffected.

GDN and KDA

If GDN uses $\alpha=0.9$ , both rows are multiplied by $0.9$ . KDA can instead choose

D=\operatorname{Diag}(0.2,0.99),

so that memory associated with $k_a$ is forgotten quickly while memory associated with $k_b$ is almost completely retained. This is the simplest illustration of the difference between a scalar and a diagonal gate.

Keys in a real model are not orthogonal, so “one row equals one independent concept” is only a teaching intuition. Part of training is learning a key-feature space that supports useful reads and writes while minimizing conflicts.

12. Kimi Linear is not pure KDA

KDA is an attention module; Kimi Linear is a complete hybrid architecture that uses it. The representative architecture in the paper interleaves KDA and MLA layers at a roughly $3{:}1$ ratio. The goal is to combine

KDA’s linear sequence complexity, fixed recurrent state, and fast long-context decoding;
full attention or MLA’s direct access to individual historical tokens, which alleviates the information bottleneck of a fixed-size state.

The paper’s claim of reducing KV-cache usage by up to 75% follows from this hybrid ratio together with MLA’s cache design. It does not mean that an individual KDA layer stores 25% of the historical KV pairs. Likewise, the reported long-context throughput gains come from the complete system: architecture, kernels, and model layout together.

13. Common misconceptions and a checklist

Misconception 1: Linear Attention is a lossless reordering of softmax attention

It is not. Only a factorable kernel permits the associative reordering. Replacing the kernel changes the model’s inductive bias and usually introduces a fixed-capacity bottleneck.

Misconception 2: Combining K and V means computing $K^\top V$ once

That description works for a non-causal, full-sequence expression. A causal model must maintain a prefix state $S_t$ so that the query at position $t$ cannot access future tokens.

Misconception 3: DeltaNet increases state capacity

It does not change the dimensions of $S_t$ . It improves how the finite capacity is used, especially by overwriting stale associations instead of accumulating them indefinitely.

Misconception 4: $\alpha$ and $\beta$ are the same kind of gate

They are not. $\alpha$ controls retention of the old state; $\beta$ controls the strength of the current delta correction.

Misconception 5: KDA is merely per-channel GDN

That is a useful high-level intuition, but the gate acts on the key-feature axis and its order relative to $(I-\beta k^\top k)$ matters. Matrix multiplication is not commutative:

$(I-\beta k^\top k)D$ ≠ $D(I-\beta k^\top k)$

in general. Always align row- versus column-vector conventions before comparing equations or code.

Misconception 6: Linear complexity always means faster execution

Actual speed also depends on sequence length, head size, chunk kernels, hardware utilization, state-memory bandwidth, and whether the architecture mixes in full attention. At short sequence lengths, a highly optimized FlashAttention implementation can remain very competitive.

14. The shortest summary

Softmax attention retains every historical key and value, allowing each query to select the past independently. It is expressive, but expensive in sequence length.
Linear Attention uses a factorable kernel and associativity to compress historical $(k_i,v_i)$ pairs into $S_t=\sum_i k_i^\top v_i$ , which is then read as $q_tS_t$ . This is the meaning of separating Q from K and combining K with V.
A purely additive state cannot naturally overwrite an old mapping.
DeltaNet treats the state as an online linear model and writes its prediction error:
$S_t=S_{t-1}+\beta_tk_t^\top(v_t-k_tS_{t-1}).$
Gated DeltaNet uses a scalar $\alpha_t$ to control retention of the whole memory.
KDA uses a diagonal matrix $D_t$ to control retention independently across key features:
$S_t=(I-\beta_tk_t^\top k_t)D_tS_{t-1}+\beta_tk_t^\top v_t.$
KDA’s diagonal-minus-rank-one transition is more expressive than a scalar gate, preserves the delta-rule structure, and supports an efficient specialized DPLR chunkwise algorithm.

References

Katharopoulos et al., Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention, ICML 2020.
Schlag, Irie, and Schmidhuber, Linear Transformers Are Secretly Fast Weight Programmers, ICML 2021. The fast-weight interpretation of Linear Attention and the delta rule.
Yang et al., Parallelizing Linear Transformers with the Delta Rule over Sequence Length, NeurIPS 2024. Recurrent and chunkwise DeltaNet formulations and hardware-efficient training.
Yang et al., Gated Delta Networks: Improving Mamba2 with Delta Rule, ICLR 2025. Data-dependent decay for DeltaNet.
Kimi Team, Kimi Linear: An Expressive, Efficient Attention Architecture, 2025. KDA, its specialized DPLR chunkwise algorithm, and the hybrid Kimi Linear architecture.