← Master Index
Vol. 18 Module 18.3 Lecture

Edge AI

Hardware & Model Optimization

How This Lesson Fits the Module & Volume

This module walked silicon (GPU, CUDA), compilers (TensorRT, ONNX), and shrink tools (quantization, compression). Edge AI is where those artifacts run off the datacenter: phones, PCs NPUs, Jetson, industrial gateways. Volume 06 VRAM/dtype math still holds—often as DRAM + NPU SRAM, not HBM. Volume 12 KV/FlashAttention ideas appear in miniature (short context, tiny caches) or disappear (encoder-only classifiers).

Module 18.4 starts next: if the “edge” is just a laptop calling an API, that is Tier 1—not Edge AI. Do not confuse on-device inference with API-only clients.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define Edge AI as on-device (or near-device) inference with tight power, memory, and privacy constraints.
  • Map runtimes: TFLite / LiteRT, ONNX Runtime, CoreML, TensorRT on Jetson, QNN, WebGPU/WASM.
  • Choose model classes that belong on-device vs on a GPU server vs API-only.
  • Apply INT8/INT4 + distillation as the default edge compression path.
  • List failure modes: thermal throttle, op coverage, model update, offline eval drift.
  • Hand off to Module 18.4 sizing: when “edge” is really a 3060 desktop (Tier 2), not a phone NPU.
Definition

Edge AI is inference (and rarely tiny on-device learning) executed on hardware co-located with the sensor or user—smartphone SoC NPUs, laptops NPUs, microcontrollers, NVIDIA Jetson, industrial PCs—rather than in a remote GPU cluster. Constraints dominate: watts, milliwatts, DRAM in the 4–16 GB class (or KB on MCUs), thermals, and often no CUDA. Success looks like a quantized ONNX/TFLite/CoreML graph with a hard latency SLO, not a 70B FP16 chat model.

What Actually Runs on the Edge

WorkloadFits on-device?Usual artifact
Keyword spot, VAD, tiny ASR encoderYes (MCU–phone)INT8 TFLite / microTVM
Vision detect / OCR / poseYes (NPU / Jetson)ONNX → TensorRT / CoreML / QNN
Embedding / rerank (small)OftenONNX INT8; distill from server teacher
7B INT4 chat (laptop/Jetson 8–16 GB)Borderline “fat edge”GGUF / ORT; this is close to 18.4 Tier 2
13B+ FP16, long-context agentsNo—server or APIvLLM / TRT-LLM / Module 18.1 SDKs

Runtime Map (no CUDA assumed)

Mobile SoC

  • CoreML (Apple), NNAPI / QNN (Android)
  • TFLite delegates, MediaPipe
  • Watch ANE/NPU op coverage

NVIDIA Jetson

  • TensorRT + CUDA (small cousin of 18.3.3)
  • Power modes / jetson_clocks
  • Still budget VRAM-like shared mem

PC NPU / Web

  • ORT DirectML, OpenVINO, WebGPU
  • Hybrid: NPU embed + API LLM
  • Do not pretend WebGPU is an H100

Export Path Toward the Device

Train in PyTorch → export ONNX (previous lectures) → convert/quantize for the vendor runtime. Keep dynamic shapes honest: phones hate unbounded seq. Hybrid apps call Module 18.1 APIs for the heavy LLM and keep PII features on-device.

import torch from pathlib import Path # 1) Export a tiny classifier (same ONNX contract as the ONNX lecture) from torch import nn class EdgeNet(nn.Module): def forward(self, x): return torch.softmax(self.fc(x), dim=-1) if hasattr(self, "_ready") else x # Practical pattern: torch.onnx.export(...) → encoder.onnx (see onnx.html) # 2) Quantize / convert offline — ORT static INT8 sketch: # from onnxruntime.quantization import quantize_static, QuantType # quantize_static("encoder.onnx", "encoder.int8.onnx", calibration_data_reader) # # 3) On device: ORT Mobile / TFLite / CoreML consume the quantized graph. # Jetson: trtexec --onnx=encoder.onnx --int8 --saveEngine=encoder.plan # Hybrid: on-device embed + HTTPS to OpenAI/Anthropic/Gemini (Vol. 18.1) for generation. # If the "device" is an RTX 3060 desktop, you left Edge AI — that is Module 18.4 Tier 2.

Privacy, Power, and Honesty

Edge wins when

  • PII / camera frames must not leave the device
  • Offline or flaky WAN (factory, aircraft, field)
  • Hard real-time < network RTT
  • Per-unit API cost would dominate at millions of devices

Do not call it Edge AI when

  • The app is only an API client (Tier 1)—valid, just different
  • You need 70B quality and will secretly round-trip anyway
  • Thermals throttle after 30 s and your demo was 5 s
  • Op fallback silently hits CPU and misses the SLO

Related Lectures

LectureWhy it sits beside Edge AI
ONNX / TensorRTExport + Jetson compile path
Quantization / CompressionHow models fit DRAM/NPU
Vol. 06 INT8Default edge dtype
Vol. 12 KV CacheWhy on-device LLMs stay short-context
18.1 OpenAI SDKHybrid: edge + API
18.4 Tier 1Next: laptop as API client, not NPU
Common Misconception

“Any local model is Edge AI.” A 4090 under a desk running vLLM is local serving (Module 18.4 Tiers 2–3), not a phone NPU. Second: “INT8 on-device matches FP16 server logits.” Measure on-device; vendor kernels differ. Third: “Edge means we never need the cloud.” Hybrid is the adult pattern: on-device for privacy/latency-critical slices, API/GPU for the heavy generator.

Knowledge Check

  1. Short Answer: Define Edge AI in one sentence. Answer: On-device (or near-device) inference under tight power, memory, and often privacy constraints.
  2. True/False: A laptop calling the OpenAI API is Edge AI. Answer: False—that is Tier 1 API-only; no on-device model.
  3. Multiple Choice: Typical phone runtime: (a) InfiniBand NCCL, (b) CoreML / TFLite / QNN, (c) multi-node H100. Answer: (b).
  4. Short Answer: Why are on-device LLMs usually short-context? Answer: KV cache grows with sequence length; edge DRAM cannot hold long KV plus weights.
  5. True/False: Jetson can still use TensorRT/CUDA even though it is “edge.” Answer: True (NVIDIA edge; not a phone SoC).
  6. Multiple Choice: Default compression path to the device: (a) FP64 teacher dumped raw, (b) distill + INT8/INT4 export, (c) skip export entirely. Answer: (b).
  7. Short Answer: Name one reason a demo passes and production fails on phones. Answer: Thermal throttle, CPU fallback for missing NPU ops, or background memory pressure (any).
  8. True/False: Hybrid on-device embed + cloud LLM can be a valid privacy design if embeddings are non-invertible enough for the threat model—still review PII. Answer: True, with the caveat that threat modeling is required.
  9. Multiple Choice: 70B FP16 chat belongs: (a) phone NPU, (b) server GPU / API, (c) 8-bit MCU. Answer: (b).
  10. Short Answer: What module/lecture sizes “any laptop + API key” next? Answer: Module 18.4 Tier 1 API-Only.

Key Takeaways

  • Edge AI = on-device inference under watts/DRAM/privacy—not every local GPU.
  • ONNX/TFLite/CoreML/TRT-Jetson + INT8/INT4 + distillation are the default path.
  • Vol. 12 KV still kills long-context on-device LLMs.
  • Hybrid edge + API is common; Tier 1 is API-only and comes next.
  • Continue with 18.4 Tier 1 API-Only.
Trainer’s Guide

Lab: Classify five product pitches (factory camera, consumer chat app, laptop Copilot, Jetson robot, MCU keyword spot) as Edge / Tier 1 API / Tier 2+ local GPU. Sketch one export path for the Jetson and one hybrid path for the chat app.

Discussion: When is “we run the model on-device for privacy” marketing vs a real threat model? Bring embeddings, logs, and update channels into the argument.

Recap: Edge AI runs compressed models next to the user. Continue with Module 18.4 Tier 1 API-Only.