← Master Index
Vol. 10 Module 10.1 Lecture

Value

Attention Mechanism

How This Lesson Fits the Module & Volume

Queries ask; keys match. The value is what actually flows into the attention output once weights are known. Without values, attention would only produce a distribution—interesting for analysis, useless as a layer.

This lecture completes the Q/K/V triad before the unifying Attention lecture. You will see v = x WV, why d_v can differ from d_k, and how the weighted sum of values becomes the new contextual vector.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define the value as the content vector mixed by attention weights.
  • Explain v = x WV and the output shape after weighting.
  • Distinguish values from keys conceptually and in code.
  • Write the attention output as softmax(scores) V.
  • Implement value projection and the weighted sum in PyTorch.
  • Connect values to encoder memory payloads in cross-attention.
Definition

A value is a vector v = x WV carrying the information to be aggregated. If attention weights for one query are α1, …, αS, the attention output is ∑j αj vj—a soft selection / blend of values.

Weights Select Values

After scores are scaled and softmaxed (see Attention Score and Scaled Dot-Product Attention), each query owns a distribution over key positions. Multiplying that distribution by V gathers content. High weight on position j means “copy more of vj into my output.”

SymbolRoleTypical shape (batch omitted)
QRequests(T_q, d_k)
KAddresses(T_k, d_k)
VPayloads(T_k, d_v)
α = softmax(scores)Weights(T_q, T_k)
α VOutput(T_q, d_v)

d_v Versus d_k

Often Equal

  • Many implementations set d_v = d_k.
  • Simplifies multi-head concat.
  • Common default in tutorials.

Can Differ

  • Matching dim ≠ content dim.
  • Paper notation allows d_v ≠ d_k.
  • Output width follows d_v.

Multi-Head

  • Per-head values, then concat.
  • Final W_O mixes heads.
  • See Multi-Head Attention later.

Value Path in PyTorch

import torch from torch import nn import torch.nn.functional as F B, T_q, T_k, d_model, d_k, d_v = 2, 3, 5, 64, 16, 16 q_in = torch.randn(B, T_q, d_model) kv_in = torch.randn(B, T_k, d_model) W_Q = nn.Linear(d_model, d_k, bias=False) W_K = nn.Linear(d_model, d_k, bias=False) W_V = nn.Linear(d_model, d_v, bias=False) Q, K, V = W_Q(q_in), W_K(kv_in), W_V(kv_in) scores = Q @ K.transpose(-2, -1) / (d_k ** 0.5) weights = F.softmax(scores, dim=-1) # (B, T_q, T_k) out = weights @ V # (B, T_q, d_v) print(weights.shape, out.shape) # torch.Size([2, 3, 5]) torch.Size([2, 3, 16])

Intuition Check

If weights are a one-hot on position j, the output equals vj—hard retrieval. Softmax yields soft retrieval: a blend. That blend is why attention can combine “Paris” and “France” signals when translating a related phrase, rather than copying a single encoder state blindly.

Strengths of Separate V

  • Content space free from match space.
  • Output dim controllable via d_v.
  • Clear gradient path into payloads.

Mistakes to Avoid

  • Multiplying weights by K instead of V.
  • Softmax on the wrong dimension.
  • Shape errors: T_k must align for K and V.
Common Misconception

“Attention returns the keys of the best matches.” Attention returns a weighted sum of values. Keys only helped compute the weights. If an implementation returns K or Q, it is not standard attention.

Knowledge Check

  1. Short Answer: What does the value carry? Answer: The content / payload to be aggregated into the attention output.
  2. True/False: Attention output is softmax(scores) multiplied by V. Answer: True.
  3. Multiple Choice: V must share its sequence length with: (a) Q only, (b) K, (c) the batch size only. Answer: (b).
  4. Short Answer: If weights are one-hot on j, what is the output? Answer: Exactly v_j.
  5. True/False: d_v is required to equal d_k in the original Transformer math. Answer: False—they may differ; many impls set them equal for convenience.
  6. Multiple Choice: In cross-attention, values come from: (a) decoder states, (b) encoder memory projections, (c) the tokenizer. Answer: (b).
  7. Short Answer: Give the shape of αV if α is (T_q, T_k) and V is (T_k, d_v). Answer: (T_q, d_v).
  8. True/False: Softmax runs over the d_v feature axis. Answer: False—it runs over key/value positions.
  9. Multiple Choice: Separating W_V from W_K lets the model: (a) skip softmax, (b) optimize content independently from matching, (c) remove the encoder. Answer: (b).
  10. Short Answer: Which lecture unifies Q, K, V into the full mechanism? Answer: Attention.

Key Takeaways

  • Values are payloads: the attention output is a weighted sum of V.
  • Keys decide weights; values decide content.
  • Shape discipline: K and V share T_k; output follows (T_q, d_v).
  • Next: Attention puts the full mechanism together.
Trainer’s Guide

Hands-on idea: Replace V with the identity (or with K) and ask students what breaks conceptually—then restore correct V.

Discussion prompt: When would you want d_v larger than d_k?

Recap: Values supply the content that attention weights blend. Continue with Attention.