AI coding agents

In plain language

An AI coding agent is a program that takes a job, such as fixing a bug or adding a feature to a software project, and works through it with limited help from a person. It can look through the project's files, run the tests, read the errors, change the code and try again. Many are built on large language models and sit inside a code editor, a terminal or a code-hosting platform.

The key questions are whether these agents really make people faster and whether the code they produce is safe to use. The first SWE-bench results in 2023 were low, with the best model reported at publication solving 1.96% of problems. Later work has questioned whether benchmark scores reflect problem-solving, because models may have seen benchmark content during training. Studies with human developers have given conflicting answers, and the tools change quickly, so results date quickly.

Technical explanation

Many coding agents wrap a large language model in a loop with tools for reading files, searching a repository, running shell commands and tests, and editing code. Performance is commonly measured on SWE-bench, in which a system receives a real GitHub issue from an open-source project and must produce a code change that resolves it. The original SWE-bench preprint (Jimenez and colleagues, 2023) contained 2,294 problems, and the best model reported at publication, Claude 2, resolved 1.96% of them. SWE-bench Verified is a 500-instance subset created in collaboration with OpenAI, in which human annotators reviewed each instance to make sure the problem descriptions were clear.

Benchmark scores can overstate what a system does in practice. Liang and colleagues (2025, preprint) report that, given only an issue description and no repository structure, state-of-the-art models identified the buggy file with up to 76% accuracy, and reached up to 53% on repositories not in SWE-bench. The authors say performance gains on SWE-bench Verified may be partly driven by memorisation rather than problem-solving, and they raise concerns about the validity of existing results. Field studies are harder to run. METR's randomised trial (Becker and colleagues, 2025) randomly assigned 246 tasks from 16 experienced open-source developers to conditions with or without AI tools, primarily Cursor Pro with Claude 3.5/3.7 Sonnet, and found that allowing AI increased completion time by 19% (confidence interval +2% to +39%). Developers had predicted a 24% reduction beforehand and estimated a 20% reduction afterwards. METR's February 2026 follow-up used a new design with 57 developers, 143 repositories and over 800 tasks. It estimated changes of -18% for returning developers and -4% for new developers, where negative means faster, but both intervals include zero. METR says developers are likely more sped up now than its 2025 estimates suggested, but calls the new data very weak evidence and names selection effects, including developers declining to take part, as a source of bias.

Security evidence is thinner and mostly older. Perry and colleagues (2022) found in a controlled user study that participants with access to an AI assistant based on OpenAI's codex-davinci-002 wrote significantly less secure code than those without one. A 2026 preprint by Siddiq and colleagues analysed over 33,000 curated agentic pull requests on GitHub and found 1,293 security-related ones, about 4%, which had lower merge rates and longer review latency. The authors attribute this to heightened human scrutiny. That finding shows how such changes are reviewed, not whether they contain fewer flaws.

What the evidence says

Each statement is typed by the strength of its evidence and linked to the source that supports it.

Limitations and open problems

• Benchmark scores may be inflated by memorisation of benchmark content, a concern raised in a 2025 preprint by Liang and colleagues that has not been settled.
• The 2025 METR trial covered early-2025 AI tools and 16 experienced open-source developers, so its result may not carry over to other tools or developers.
• METR calls its 2026 data very weak evidence. Its intervals include zero, and the post reports selection effects, including a rise in developers choosing not to take part.
• The 2022 security study used an earlier model in a controlled setting, and the 2026 pull-request study reports review outcomes rather than the rate of flaws in agent-written code.
• Developers' own estimates of their speed did not match measured results in the 2025 trial, so self-reported gains are not a reliable guide to real productivity.

Possible future uses

Timeline

1. Oct 2023The SWE-bench preprint introduces 2,294 problems from real GitHub issues; the best model reported, Claude 2, resolves 1.96% of them. Sources: [1]
2. Jun 2025Liang and colleagues post a preprint reporting that gains on SWE-bench may be partly driven by memorisation, which raises validity concerns. Sources: [3]
3. Jul 2025METR publishes its randomised trial finding that experienced developers took 19% longer with AI tools allowed. Sources: [4]
4. Jan 2026Siddiq and colleagues post a preprint on 1,293 security-related agentic pull requests on GitHub. Sources: [7]
5. 24 Feb 2026METR reports a new-design follow-up with 57 developers, says developers are likely more sped up now than its 2025 estimates, and notes that its new data is very weak evidence. Sources: [5]

Futures this could help bring about

How far away?

Most software written by AI

Scaling up · 1 of 6 steps done

Glossary

Coding agent

An AI system that plans and carries out software tasks, such as reading code, editing files and running tests, with limited human direction.

SWE-bench

A benchmark in which a system must produce a code change that resolves a real GitHub issue from an open-source project.

Randomised controlled trial

A study in which participants or tasks are randomly assigned to conditions, so differences between groups can be attributed to the condition being tested.

Confidence interval

A range of values that a measured estimate is consistent with. If the range includes zero, the study has not shown a clear effect in either direction.

Sources

1. preprint
   SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
   Carlos E. Jimenez, John Yang, Alexander Wettig et al.. arXiv, Oct 2023. Accessed 21 Oct 2026.
2. other
   SWE-bench Verified
   SWE-bench project, 2026. Accessed 21 Oct 2026.
3. preprint
   The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason
   Shanchao Liang, Spandan Garg, Roshanak Zilouchian Moghaddam. arXiv, Jun 2025. Accessed 21 Oct 2026.
4. preprint
   Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
   Joel Becker, Nate Rush, Elizabeth Barnes et al.. arXiv, Jul 2025. Accessed 21 Oct 2026.
5. other
   We are Changing our Developer Productivity Experiment Design
   Joel Becker, Nate Rush, Tom Cunningham et al.. METR, 24 Feb 2026. Accessed 21 Oct 2026.
6. preprint
   Do Users Write More Insecure Code with AI Assistants?
   Neil Perry, Megha Srivastava, Deepak Kumar et al.. arXiv, Nov 2022. Accessed 21 Oct 2026.
7. preprint
   Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub
   Mohammed Latif Siddiq, Xinye Zhao, Vinicius Carvalho Lopes et al.. arXiv, Jan 2026. Accessed 21 Oct 2026.

Revision history

3 changes since this profile was created.