BERT and RoBERTa scale by stacking wider, deeper encoders—and parameter counts explode. ALBERT (A Lite BERT, Lan et al., 2019) attacks parameter efficiency: factorized embeddings and cross-layer weight sharing, plus a sentence-order objective instead of classic NSP.
It bridges “make it better” (RoBERTa) and “make it smaller / cheaper” (DistilBERT, ELECTRA).
Learning Objectives
By the end of this lesson, students should be able to:
- Explain factorized embedding parameterization.
- Describe cross-layer parameter sharing and its effect on size vs. compute.
- Contrast Sentence Order Prediction (SOP) with NSP.
- Load an ALBERT checkpoint in Hugging Face.
- State when ALBERT helps memory-bound deployments.
- Note that fewer unique parameters ≠ necessarily fewer FLOPs per forward.
ALBERT is a BERT-style encoder that reduces unique parameters by (1) projecting vocabulary embeddings through a smaller factorized matrix and (2) sharing Transformer weights across layers, trained with MLM and Sentence Order Prediction.
Two Parameter Tricks
Factorized Embeddings
- Vocab × E, then E → H
- Decouples vocab size from hidden size
- Cuts embedding matrix cost
Cross-Layer Sharing
- Same block weights reused L times
- Far fewer unique params
- Depth without linear param growth
SOP Objective
- Two consecutive sentences
- Positive: correct order
- Negative: swapped order
SOP vs. NSP
NSP mixes topic prediction (different documents) with discourse continuity and is often too easy. SOP always uses two sentences from the same document; the model must detect whether order was swapped—a finer coherence signal.
| Model | Unique params (approx.) | Notes |
|---|---|---|
| BERT-Base | ~110M | No sharing |
| ALBERT-Base | ~12M | Sharing + factorization |
| ALBERT-xxlarge | ~235M unique | Very wide; still shared depth |
“Fewer parameters always means faster inference.” Layer sharing reduces storage of weights, but you still run L layer computations. Latency can remain similar to a deep BERT unless you also reduce depth or width.
Code
Strengths and Tradeoffs
Strengths
- Much smaller checkpoint footprint.
- SOP is a cleaner discourse task than NSP.
- Enables wider models under param budgets.
Tradeoffs
- Sharing can limit representational flexibility.
- FLOPs per token may still be high.
- Ecosystem slightly thinner than BERT/RoBERTa.
Knowledge Check
- Short Answer: What does ALBERT stand for? Answer: A Lite BERT.
- True/False: Factorized embeddings let vocab size grow without a full vocab×H matrix. Answer: True.
- Multiple Choice: Cross-layer sharing means: (a) layers share optimizer state only, (b) the same weights are reused across depth, (c) no attention. Answer: (b).
- Short Answer: How does SOP create negatives? Answer: Swap the order of two consecutive sentences.
- True/False: Fewer unique parameters always imply proportionally fewer FLOPs. Answer: False.
- Multiple Choice: ALBERT replaces classic NSP with: (a) SOP, (b) RLHF, (c) CTC. Answer: (a).
- Short Answer: Name one deployment benefit of ALBERT. Answer: Smaller weight storage / memory footprint.
- Short Answer: Why can NSP be “too easy”? Answer: Topic mismatch between random sentences is a strong cue.
- Multiple Choice: ALBERT is still: (a) encoder-only NLU style, (b) a diffusion U-Net, (c) purely rule-based. Answer: (a).
- True/False: ALBERT keeps MLM as a core pretraining signal. Answer: True.
Key Takeaways
- ALBERT shrinks unique params via factorization + layer sharing.
- SOP improves on NSP by testing sentence order, not topic.
- Parameter count ≠ latency; FLOPs can remain large.
- Useful when memory and checkpoint size dominate constraints.
- Next: DistilBERT compresses via distillation.
Hands-on idea: Print parameter counts for bert-base-uncased vs. albert-base-v2; discuss storage vs. latency.
Discussion prompt: Would you rather share layers or distill a student (DistilBERT) for a mobile app?
Recap: ALBERT is BERT made parameter-lite. Continue with DistilBERT.