← Master Index
Vol. 10 Module 10.1 Lecture

Attention Score

Attention Mechanism

How This Lesson Fits the Module & Volume

The Attention lecture gave the full pipeline. This lesson zooms into the middle step: the attention score—the raw compatibility between a query and a key before softmax.

Different scoring functions exist (additive / Bahdanau, multiplicative / Luong, dot-product). Transformers standardize on scaled dots, covered next in Scaled Dot-Product Attention. Understanding scores first makes that scaling feel motivated, not magical.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define an attention score as a scalar compatibility between one query and one key.
  • Compare additive (Bahdanau) and dot-product (Luong / Transformer) scoring.
  • Explain the role of masks as score surgery (−∞ before softmax).
  • Show how score matrices have shape (T_q, T_k) per batch item.
  • Implement several score functions in PyTorch and verify softmax inputs.
  • State why large-magnitude scores are problematic (leading into scaling).
Definition

An attention score eij = score(qi, kj) is an unnormalized measure of how well key j matches query i. Softmax over j converts the row of scores into attention weights αij.

Common Scoring Functions

NameFormula (sketch)Notes
Dot-productq · kFast; needs similar scales
Scaled dot-product(q · k) / √d_kTransformer default
Multiplicative (Luong general)qT W kLearned bilinear map
Additive (Bahdanau)vT tanh(W_q q + W_k k)Classic RNN attention

From Scores to Weights

Score Matrix

  • Shape (B, T_q, T_k).
  • Can be any real numbers.
  • Larger ⇒ preferred before softmax.

Masking

  • Set invalid scores to −∞.
  • PAD, future tokens, etc.
  • Softmax weight → 0 there.

Softmax Row-wise

  • Normalize over keys (dim −1).
  • Each query gets a distribution.
  • Then multiply by V.

Implementing Scores in PyTorch

import torch from torch import nn import torch.nn.functional as F B, T_q, T_k, d_k = 2, 3, 4, 8 Q = torch.randn(B, T_q, d_k) K = torch.randn(B, T_k, d_k) # Dot-product scores scores_dot = Q @ K.transpose(-2, -1) # (B, T_q, T_k) # Additive (Bahdanau-style) scores W_q = nn.Linear(d_k, d_k, bias=False) W_k = nn.Linear(d_k, d_k, bias=False) v = nn.Linear(d_k, 1, bias=False) q_p = W_q(Q).unsqueeze(2) # (B, T_q, 1, d_k) k_p = W_k(K).unsqueeze(1) # (B, 1, T_k, d_k) scores_add = v(torch.tanh(q_p + k_p)).squeeze(-1) # Mask out last key position as PAD mask = torch.ones(B, T_q, T_k) mask[:, :, -1] = 0 scores_dot = scores_dot.masked_fill(mask == 0, float("-inf")) weights = F.softmax(scores_dot, dim=-1) print(scores_dot.shape, weights[0, 0]) # last weight ~ 0

Why Magnitude Matters

Dot products grow with dimension: if components are roughly variance 1, q · k has variance about d_k. Large scores push softmax into a near one-hot regime with tiny gradients. That is exactly why the next lecture divides by √d_k.

Dot-Product Pros

  • One matmul—highly optimized.
  • No extra score parameters.
  • Natural multi-head packing.

Additive Pros

  • Flexible learned compatibility.
  • Historically strong with RNNs.
  • Can be stabler at small d_k sometimes.
Common Misconception

“Attention scores are already probabilities.” Scores are free real numbers. Only after softmax (and masking) do you get a distribution that sums to one across keys.

Knowledge Check

  1. Short Answer: What is an attention score e_ij? Answer: An unnormalized compatibility between query i and key j.
  2. True/False: Softmax turns scores into attention weights. Answer: True.
  3. Multiple Choice: Bahdanau attention uses: (a) only q·k, (b) an additive tanh scoring MLP, (c) convolution. Answer: (b).
  4. Short Answer: How do you mask a forbidden key in the score matrix? Answer: Set its score to −∞ before softmax.
  5. True/False: Score matrices have shape (B, T_q, T_k) for batched Q, K. Answer: True.
  6. Multiple Choice: Softmax for attention runs over: (a) feature dim d_k, (b) key positions, (c) batch only. Answer: (b).
  7. Short Answer: Why can large d_k make raw dot products problematic? Answer: Scores grow in magnitude, softmax saturates, gradients shrink.
  8. True/False: Attention scores are the same object as values. Answer: False.
  9. Multiple Choice: Luong “general” scoring introduces: (a) a bilinear W, (b) dropout only, (c) pooling. Answer: (a).
  10. Short Answer: What factor does scaled dot-product attention divide by? Answer: Square root of d_k.

Key Takeaways

  • Scores measure query–key compatibility before softmax.
  • Dot-product, scaled dot-product, multiplicative, and additive variants exist.
  • Masks edit scores; softmax yields weights over keys.
  • Next: Scaled Dot-Product Attention—the Transformer workhorse.
Trainer’s Guide

Hands-on idea: Compare weight entropy for raw vs. scaled dots as d_k increases (8 → 64 → 256).

Discussion prompt: When might you still prefer Bahdanau additive attention over dots?

Recap: Attention scores are raw match strengths; softmax and masks turn them into usable weights. Continue with Scaled Dot-Product Attention.