IA3

How This Lesson Fits the Module & Volume

LoRA adds low-rank matrices; IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations) multiplies key, value, and feed-forward activations by learned vectors. Even fewer parameters than LoRA, with strong multi-task results in the PEFT literature—another point on the spectrum between full FT and tiny prompts.

Learning Objectives

By the end of this lesson, students should be able to:

Define IA3 as learned rescaling of activations.
Identify typical injection sites (K, V, FFN).
Compare parameter count vs LoRA and prompt tuning.
Configure IA3 with Hugging Face PEFT.
Discuss multi-task IA3 adapter sharing.
Choose IA3 when extreme PEFT lightness matters.

Definition

IA3 adapts a frozen Transformer by elementwise-multiplying selected activation streams with trainable vectors l_k, l_v, l_ff (and similar). Instead of adding ΔW, it rescales inner activations—inhibiting or amplifying channels that matter for the task.

Mechanism

Freeze W

Base Transformer fixed

Scale K/V

Attention activations × vectors

Scale FFN

Intermediate activations × vectors

Train vectors

Tiny PEFT footprint

Method	What is trained	Footprint feel
Full FT	All weights	Huge
LoRA	Low-rank A, B	Small
IA3	Rescaling vectors	Tiny
Prompt tuning	Soft prompt embeddings	Tiny (input-side)

PEFT IA3 Example

from transformers import AutoModelForSeq2SeqLM
from peft import IA3Config, get_peft_model, TaskType

# IA3 was popularized on T5-style models; also available for causal LMs in PEFT
model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
cfg = IA3Config(
    task_type=TaskType.SEQ_2_SEQ_LM,
    target_modules=["k", "v", "wi"],  # names depend on architecture
    feedforward_modules=["wi"],
)
model = get_peft_model(model, cfg)
model.print_trainable_parameters()
# Train with standard seq2seq / SFT loops; export tiny IA3 vectors per task

LoRA vs IA3

LoRA

Additive low-rank update.
More capacity per layer.
Default for LLM SFT.

Multiplicative rescaling.
Fewer parameters.
Strong multi-task papers.

Pick IA3 when

Extreme storage limits.
Many task vectors.
Rescaling suffices.

Strengths and Tradeoffs

Strengths

Very small adapter files.
Stable, simple structure.
Composable multi-task setups.

Tradeoffs

Less expressive than high-rank LoRA.
Module name mapping is arch-specific.
Less common than LoRA/QLoRA in LLM ops.

Common Misconception

“IA3 is another name for LoRA.” LoRA adds a low-rank residual to weights; IA3 multiplies activations by learned vectors. Different inductive bias, different capacity.

Knowledge Check

Short Answer: What does IA3 learn? Answer: Rescaling vectors for selected activations (e.g., K, V, FFN).
True/False: IA3 typically trains full d×d weight matrices. Answer: False.
Multiple Choice: IA3 updates are primarily: (a) multiplicative, (b) convolutional, (c) dropout-only. Answer: (a).
Short Answer: Compared to LoRA, IA3 parameter count is usually: Answer: Smaller / tinier.
True/False: The base Transformer stays frozen in IA3. Answer: True.
Multiple Choice: PEFT config class: (a) IA3Config, (b) MaxPool2d, (c) BeamConfigCSS. Answer: (a).
Short Answer: Name one typical injection site. Answer: Keys, values, or feed-forward activations.
True/False: IA3 and LoRA use identical inductive biases. Answer: False.
Multiple Choice: IA3 is attractive for: (a) many tiny task packs, (b) only CNNs, (c) DNS. Answer: (a).
Short Answer: What soft-prompt method comes next? Answer: Prefix tuning.

Key Takeaways

IA3 rescales activations with learned vectors.
Ultra-light PEFT alternative to LoRA.
Great for multi-task vector packs; less default for big LLM SFT.
Still freezes the pretrained backbone.
Next: Prefix Tuning.

Trainer’s Guide

Hands-on idea: Print trainable param counts for LoRA r=8 vs IA3 on the same T5; discuss capacity vs storage.

Discussion prompt: For 100 enterprise tasks, would you rather store 100 LoRA or 100 IA3 packs?

Recap: IA3 adapts models by amplifying or inhibiting activations with tiny vectors. Continue with Prefix Tuning.