Kubernetes — Module 18.2

How This Lesson Fits the Module & Volume

Docker gave you one reproducible API container. Kubernetes (K8s) is how teams run many copies: rollouts, health probes, secrets, and (later) GPU node pools for self-host infer. Your FastAPI wrapper from Module 18.1 SDKs usually starts as a Deployment + Service + Ingress. You do not need K8s to call OpenAI; you need it when uptime, scale, and multi-service wiring (Redis, Celery workers) outgrow Compose on one VM.

Next: Redis as the cache/broker those workers share. Module 18.3 GPU lectures will revisit nvidia.com/gpu resource requests.

Learning Objectives

By the end of this lesson, students should be able to:

Map Pod, Deployment, Service, Ingress, ConfigMap, and Secret to an AI API.
Write a minimal Deployment YAML that runs the Dockerized FastAPI image.
Use liveness/readiness probes on /health so bad rollouts stop taking traffic.
Inject vendor API keys via Secrets—not plaintext in YAML committed to git.
Contrast Compose vs K8s vs “just a VM + systemd.”
Preview GPU scheduling without treating K8s as a training framework.

Definition

Kubernetes is a container orchestrator: a control plane watches desired state (YAML/Helm) and reconciles actual cluster state. A Pod is the smallest unit (one or more containers sharing network/volumes). A Deployment manages replica Pods and rolling updates. A Service gives a stable DNS/IP; Ingress (or Gateway) terminates HTTP. K8s does not replace your API design or your SDK.

Object Cheat Sheet for This Volume

Object	AI backend job
Deployment	N replicas of `chat-api:0.1.0` (Uvicorn)
Service	Cluster DNS `chat-api.default.svc` port 8000
Ingress / Gateway	Public HTTPS → Service; TLS here, not in Flask
Secret	`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`
ConfigMap	Non-secret: default model SKU, log level
HPA	Scale replicas on CPU or custom queue depth
Job / CronJob	Batch embeddings, evals (Vol. 19)

Minimal Deployment Sketch

# k8s/chat-api.yaml  (teaching sketch — not a full prod chart)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chat-api
spec:
  replicas: 2
  selector:
    matchLabels: { app: chat-api }
  template:
    metadata:
      labels: { app: chat-api }
    spec:
      containers:
        - name: api
          image: ghcr.io/example/chat-api:0.1.0
          ports: [{ containerPort: 8000 }]
          env:
            - name: OPENAI_API_KEY
              valueFrom: { secretKeyRef: { name: llm-keys, key: openai } }
            - name: REDIS_URL
              value: redis://redis:6379/0
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            initialDelaySeconds: 5
          livenessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 20
          resources:
            requests: { cpu: "100m", memory: "256Mi" }
            limits:   { cpu: "1", memory: "1Gi" }
---
apiVersion: v1
kind: Service
metadata: { name: chat-api }
spec:
  selector: { app: chat-api }
  ports: [{ port: 80, targetPort: 8000 }]

# kubectl apply -f k8s/chat-api.yaml
# kubectl create secret generic llm-keys --from-literal=openai="$OPENAI_API_KEY"
# GPU infer later: resources.limits["nvidia.com/gpu"] = 1  (Module 18.3)

When to Graduate from Compose

Stay on Compose / one VM

Solo demo, class lab
Single region, low traffic
You still need Docker images

Move to K8s

Rolling deploys + probes
API + Redis + Celery at scale
GPU pools, multi-team clusters

Not K8s’s job

Prompt quality (Vol. 13–15)
Sampler math (Vol. 17)
Replacing authentication design

K8s buys

Desired-state ops, self-heal
Standard secrets/config/ingress
Same image from Docker lecture

K8s costs

YAML/Helm complexity
Wrong probes = flapping outages
GPU sharing still needs device plugins

Related Lectures

Lecture	Role
Docker	Image you schedule
Redis / Celery	More Deployments in the same namespace
Deployment	Envs, rollouts, GitOps story
GPU	`nvidia.com/gpu` requests

Common Misconception

“Kubernetes hosts the LLM.” It hosts processes. The model may still be OpenAI. Second: replicas: 10 on a stateful GPU infer Pod without a queue just OOMs ten times. Third: putting keys in Deployment YAML “just for class” trains the worst habit in the volume. Fourth: Ingress is not authentication—anyone who hits the URL still needs auth.

Knowledge Check

Short Answer: What is the smallest schedulable unit in K8s? Answer: A Pod.
True/False: A Deployment manages replica Pods and rolling updates. Answer: True.
Multiple Choice: Vendor keys should live in: (a) Secret, (b) committed YAML, (c) the system prompt. Answer: (a).
Short Answer: Why a readiness probe on /health? Answer: So unready Pods are removed from Service traffic during start/fail.
True/False: Kubernetes replaces the need for Docker images. Answer: False—it runs images.
Multiple Choice: GPU infer typically requests: (a) nvidia.com/gpu, (b) CFG scale, (c) t-SNE. Answer: (a).
Short Answer: Service vs Ingress? Answer: Service is in-cluster load balancing/DNS; Ingress exposes HTTP(S) from outside.
True/False: HPA can scale chat-api replicas independently of Celery workers. Answer: True—separate Deployments.
Multiple Choice: Next lecture: (a) Redis, (b) Midjourney, (c) ElasticNet. Answer: (a).
Short Answer: When is Compose enough vs K8s? Answer: Labs/single VM vs multi-service scale, probes, rolling deploys, GPU pools.

Key Takeaways

K8s reconciles desired state: Pods, Deployments, Services, Ingress.
Same Docker image; Secrets for SDK keys; probes on /health.
Compose first; K8s when ops complexity is justified.
GPU resources are Module 18.3; queues are Redis/Celery next.
Next: Redis.

Trainer’s Guide

Lab: Kind/minikube: apply the sketch (dummy secret). Break /health and watch the Service stop routing. Scale replicas 1→3.

Whiteboard: Client → Ingress → Service → Pods → OpenAI. Draw Redis/Celery as extra Deployments sharing a namespace.

Recap: Kubernetes runs your containers at scale. Add a shared data plane next: Redis.