Junior devs who learnt a new library with AI scored 50% on the follow-up quiz. The ones who hand-coded it scored 67% (and the AI group wasn't even significantly faster).

That's from an Anthropic randomised trial with 52 mostly junior engineers.

It's probably the most useful thing a CS student can read about software engineering with AI coding agents this year.

Because the agents aren't going anywhere.

When METR tried to rerun its developer study, lots of developers opted out because they "do not wish to work without AI".

So the question isn't whether to use them. It's how to use them without outsourcing your learning.

How AI coding agents are actually used in real teams

What the evidence says right now:

  1. Feelings lie. In METR's 2025 study, 16 experienced open-source developers took 19% longer with AI, yet still believed it had sped them up by 20%
  2. It's probably better now, nobody knows by how much. METR's 2026 update says AI likely speeds developers up more than in early 2025, but the data is weak, partly because 30% to 50% of developers skipped some tasks rather than do them without AI
  3. Maintenance is the hidden bill. TechCrunch covered Singapore Management University research finding AI-generated code "can introduce long-term maintenance costs", with advice to review AI output as if a junior wrote it
  4. Agents are an attack surface. In June, Tenet Security showed a fake Sentry error could hijack Claude Code, Cursor and Codex through MCP, with 85% success across 100+ targets and 2,388 organisations exposed (report). These are the vendor's own test figures, not independently verified

The AI coding toolkit worth a CS student's time

Keep it boring and small:

→ one terminal agent: Claude Code or OpenAI's Codex CLI. Great for multi-file changes, and for learning to give precise instructions → one AI editor: Cursor, or GitHub Copilot in VS Code. Good for inline help whilst you still write most of the code yourself → git, properly: branches, small commits, and reading git diff before every commit. This is your safety net → a test runner: pytest, Jest, whatever your language uses. Tests are how you judge code you didn't write

(check what student plans exist before paying for anything, they change a lot)

What uni won't teach you about AI coding agents

If your course bans AI on coursework, or just ignores it, nobody teaches you:

  1. how to write a spec an agent can actually act on
  2. how to review a diff you didn't write
  3. how to sandbox a tool that can run commands on your machine

The Anthropic study shows why the habit matters more than the tool:

bad: delegate everything (vibe coding style), or use AI as a debugging crutch (those participants averaged under 40%) good: ask conceptual questions, or ask for explanations alongside the code (those averaged 65% or more)

Same tool. Different habit. Very different learning.

A critical-thinking framework before you merge AI code

Five questions, every single time:

  1. Did I write the spec first? Acceptance criteria or a failing test before the prompt. If you can't define "done", the agent will define it for you
  2. Can I explain every line? If not, ask the agent WHY, not just what. That's the habit that kept quiz scores up
  3. What test would fail if this were wrong? If there isn't one, write it. The biggest gap in the Anthropic study was debugging
  4. What did the agent read, and do I trust it? Logs, issues, web pages and MCP tools are all input. Treat them like user input: untrusted. Don't auto-approve shell commands
  5. Would I approve this PR from a stranger? Because honestly, that's what it is

What to do this month

  1. Build one small project twice: once by hand, once with an agent. Compare the time AND how well you can explain each one
  2. Time yourself honestly for a week. METR showed your gut will overestimate the speed-up
  3. Set your agent up with least privilege: no auto-run of shell commands, no real secrets in the repo, a separate folder or container
  4. Review one open-source pull request a week. Reading code you didn't write is the skill agents make more valuable, not less

so TLDR: use the agents, but make them teach you, test everything they write, and treat everything they read as untrusted.

What's one thing you'd never let an AI agent do on your machine?

Sources

Get the next issue by email

theVANTAGEd is written by students to bring critical thinking to AI across real industries. One issue every two weeks.