Coding Assistants — Module 21.1

How This Lesson Fits the Module & Volume

Vol. 21 so far built conversation and ops products: chatbots, support, search, documents, voice, email, and workflows. Coding assistants are the same product family pointed at software work: inline completion, chat-in-IDE, and repo-aware edit suggestions. The developer is almost always human-in-the-loop—the model proposes; the human applies, tests, and owns the diff.

This lecture is product engineering: context windows, retrieval into the editor, and HumanEval-style eval conceptually (Vol. 19 benchmarks). It is not a guide to hacking, exploiting, jailbreaking, or writing malware. Vol. 20 security stays defensive. Vendor IDEs themselves appear in Vol. 22.6 (GitHub Copilot, Cursor, and siblings). Next lecture: research assistants—citation instead of unit tests.

Learning Objectives

By the end of this lesson, students should be able to:

Define a coding assistant as an HITL IDE product, not an unsupervised agent that writes production without review.
Name the main UX patterns: ghost text, chat, multi-file edit, and test-driven repair.
Budget a context window (system, cursor, tabs, repo map, retrieval, decode reserve) using Vol. 11 context length.
Explain HumanEval-style pass@k conceptually and why internal unit-test suites beat leaderboard cosplay.
List product metrics beyond pass@k: accept rate, revert rate, test-pass after apply, latency.
Refuse exploit / malware / unauthorized-access use cases in the product policy (Vol. 20).

Definition

A coding assistant is an AI product embedded in a development environment that proposes code, explanations, or edits conditioned on developer-visible context (open file, cursor, selection, optional repo retrieval) and that requires a human to accept, reject, or revise before the change is real. Ghost text is inline completion. A repo-aware assistant retrieves or summarizes other files when they do not fit in the context window. HumanEval-style eval means: generate candidate functions from a spec and score whether hidden unit tests pass (pass@k)—a category of code eval from Vol. 19, not a license to claim a public leaderboard number.

IDE Copilot Patterns

Pattern	Context the model sees	Human action	Typical risk
Ghost-text completion	Current file + nearby lines + light language ID	Tab accept / ignore	Plausible but wrong APIs; license-tainted snippets
Inline chat / “edit this”	Selection + user instruction + maybe open tabs	Review diff, run tests	Over-eager refactors; missed call sites
Repo Q&A	Retrieved chunks + file tree summary	Verify against source	Stale index; hallucinated paths (Vol. 19)
Multi-file edit plan	Plan + targeted files (budgeted)	Apply file-by-file	Partial apply; broken build
Test-driven repair	Failing test output + relevant source	Confirm tests go green	Teaching to the test; skipping real bugs

All five are copilot modes in the Vol. 15 HITL table: the human drives; the model suggests; unsupervised prod writes are out of scope. Pair with Vol. 13 guardrails for policy (no secrets in prompts, no generating exploit PoCs) and Vol. 20 copyright for training/output hygiene.

Context Windows in the Editor

Vol. 11 taught that context length is a hard token budget. In an IDE the budget is contested by many claimants. Dumping the whole repo into the prompt is not a strategy; it is a latency and “lost in the middle” failure. Retrieve, rank, and reserve decode tokens.

Always include

System / tool policy (short)
Cursor neighborhood + selection
Language, formatter, test command
Decode reserve (completion tokens)

Often include

Open tabs (capped)
Repo map (paths + one-line summaries)
Retrieved sibling files (Vol. 14 RAG)
Failing test excerpt (repair mode)

Usually exclude

Lockfiles, minified bundles, node_modules
Secrets, .env, credentials
Unrelated megabytes “just in case”
Binary / generated artifacts

Tight context buys

Lower latency and cost (Vol. 19 latency, tokens)
Less distraction / lost-in-the-middle
Clearer eval: you know what the model saw

Naïve dump costs

Timeouts and truncated prompts
Suggestions that ignore the cursor
Accidental secret exfiltration to a vendor log (Vol. 20 privacy)

Eval: HumanEval-Style, Conceptually

Public HumanEval / MBPP-style suites ask a model to synthesize a small function from a docstring and score pass@k: among \(k\) samples, did any pass all unit tests? Vol. 19 benchmarks already warned: that contract is small-function synthesis, not repo-scale engineering, security review, or style. A coding product still needs an internal harness: tasks drawn from your public tests, plus product telemetry.

Signal	What it measures	What it misses
pass@k on internal tests	Functional correctness under the test contract	Missing tests, flaky tests, security, UX
Accept rate	Whether developers trust ghost text	Accepted-but-wrong code
Revert / undo rate	Suggestions that did not survive	Silent bugs that were not reverted
CI green after apply	Integration health	Non-CI properties (a11y, cost)
Human eval (Vol. 19)	Readability, fit to house style	Expensive; sample carefully

Context Budget + pass@k Sketch

Educational product code for systems you maintain: budget tokens and score internal unit tests. Not an exploit, not a CTF solver, not instructions to attack any system.

# IDE copilot: budget context; evaluate with unit-test pass@k (HumanEval family).
# Educational only. Do not use this to attack, exploit, or bypass security.

from dataclasses import dataclass

@dataclass
class ContextBudget:
    max_tokens: int
    system: int = 800
    cursor_snippet: int = 1200
    open_tabs: int = 2000
    repo_map: int = 1500
    retrieval: int = 2000
    decode_reserve: int = 512

    def remainder_for_chat(self) -> int:
        used = (self.system + self.cursor_snippet + self.open_tabs
                + self.repo_map + self.retrieval + self.decode_reserve)
        return max(0, self.max_tokens - used)

def rank_files(query_terms: str, file_summaries: list, k: int = 8):
    """Keyword overlap over *summaries you already index* — not a secret dump."""
    q = set(query_terms.lower().split())
    scored = []
    for path, summary in file_summaries:
        words = set(summary.lower().split())
        scored.append((len(q & words), path))
    scored.sort(reverse=True)
    return [p for s, p in scored[:k] if s > 0]

def pass_at_k(n: int, c: int, k: int) -> float:
    """Unbiased pass@k: n samples, c passing all tests (Vol. 19 benchmarks)."""
    if n < k:
        raise ValueError("need n >= k")
    if n - c < k:
        return 1.0
    num = 1.0
    for i in range(k):
        num *= (n - c - i) / (n - i)
    return 1.0 - num

POLICY = {
    "human_must_accept_diff": True,
    "forbid_exploit_or_malware_tasks": True,
    "redact_env_and_secrets": True,
    "vendor_no_train_on_private_repo": True,
}
# Product dashboard: accept_rate, revert_rate, ci_green_after_apply, p95_latency.

Related Lectures

Lecture	Role
Workflow automation	Previous sibling: ops graphs vs IDE loops
Research assistants	Next: citation instead of unit tests
Document AI / AI search	Repo Q&A reuses retrieval UX
Context length (11.4)	Token budget physics
HITL / Guardrails	Accept/reject + policy
Benchmarks	HumanEval-style pass@k category
Security / Copyright	Defensive + license hygiene
Vol. 22.6 Copilot / Cursor	Vendor ecosystem later

Common Misconception

“If HumanEval pass@k is high, the assistant is production-ready.” Small-function synthesis \(\neq\) multi-file design, tests, or security. Second: stuffing the whole monorepo into context improves quality—usually it hurts. Third: accepted ghost text is correct code. Fourth: a coding assistant should “just run” unsupervised on production. Fifth: using the product to generate exploits or malware is a clever eval—it is a Vol. 20 policy failure. Sixth: public leaderboard numbers copied from memory belong on your product card (Vol. 19 said they do not).

Knowledge Check

Short Answer: What makes a coding assistant HITL rather than an unsupervised agent? Answer: A human must accept/reject/revise before the change is real.
True/False: Ghost text is inline completion in the editor. Answer: True.
Multiple Choice: HumanEval-style scoring is typically: (a) pass@k / unit tests, (b) wiki perplexity only, (c) TTFT only. Answer: (a).
Short Answer: Name three claimants on an IDE context budget. Answer: Any of: system policy, cursor/selection, open tabs, repo map, retrieval, decode reserve.
True/False: Dumping the entire repo into the prompt is the recommended context strategy. Answer: False—budget, retrieve, and reserve decode tokens.
Multiple Choice: Accept rate without revert/CI metrics mainly shows: (a) developer trust/usage, (b) formal proof of correctness, (c) GDPR certification. Answer: (a).
Short Answer: Which Vol. 11 lecture bounds how much editor context you can send? Answer: Context length (context window).
True/False: This lecture teaches how to generate exploits or attack systems. Answer: False—it is defensive product engineering only.
Multiple Choice: Repo Q&A without retrieval when files exceed the window will often: (a) hallucinate paths/APIs, (b) lower token cost automatically, (c) replace unit tests. Answer: (a).
Short Answer: Name one Vol. 21 sibling that also uses retrieval UX. Answer: AI search or Document AI (either).

Key Takeaways

Coding assistants are HITL IDE products: propose, human owns the diff.
Context is a budgeted window (Vol. 11), not a whole-repo paste.
HumanEval-style pass@k is a code-eval category; ship internal tests + accept/revert/CI metrics.
Policy forbids exploit/malware tasks; secrets stay out of prompts (Vol. 20).
Next: Research assistants — citations and RAG faithfulness.

Trainer’s Guide

Lab: Students design a context budget for a 32k-token window and a 5-task internal pass@k set from a toy calculator module they own. Measure accept vs revert on a 15-minute pairing session. No attacking third-party systems; no malware tasks.

Whiteboard: Draw ghost text vs chat vs multi-file plan. Arrow “HumanEval pass@k” to a small function box and a big X over “therefore the IDE is done.” Preview Vol. 22.6 vendor tools as examples, not homework to reverse-engineer.

Recap: Coding assistants budget editor context, keep humans on the diff, and evaluate with HumanEval-style tests plus product telemetry—never as an exploit toolkit. Continue to Research assistants.