Does `AGENTS.md` Actually Help Coding Agents? — ETH Zürich Paper Analysis

When using coding agents, you might encounter files like AGENTS.md, CLAUDE.md, or CODEX.md — these are context files. They serve as a kind of "README for agents," informing them about the repository's structure, build process, coding conventions, and so on. All agent vendors, including Anthropic, OpenAI, and Alibaba, actively recommend them, and they are included in over 60,000 repositories on GitHub alone.

But are they really effective?

The paper "Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?" by Thibaud Gloaguen et al. from ETH Zürich, presented at ICML 2026, provides the first quantitative answer to this question. In short, context files in their current form are not as effective as expected.

Experiment Design: Two Benchmarks, Three Conditions

The researchers used two benchmarks:

• SWE-bench Lite: 300 GitHub issues extracted from 11 popular Python projects. These repositories do not have context files.
• AGENTbench (new): A benchmark built by the researchers themselves. It collected 138 issues from 12 niche repositories where developers had actually committed context files.

In each benchmark, they experimented with three conditions:

1. None: No context file
2. LLM: Context file automatically generated by the agent's /init command
3. Human: Context file written directly by a developer (AGENTbench only)

The coding agents tested were Claude Code(Sonnet 4.5), Codex(GPT-5.2, GPT-5.1 mini), and Qwen Code(Qwen3-30b-coder).

Key Results: LLM-Generated Files Are Counterproductive

Here's a summary of the results:

LLM-generated context files:

• Performance decrease in 5 out of 8 settings (average -0.5% to -2%)
• Inference cost increased by 20-23%
• Number of steps increased by an average of 2.5-3.9

Context files written directly by developers:

• Slight performance improvement in 3 agents except for Claude Code (average +4%)
• However, costs also increased (up to 19%)

In other words, LLM-generated context files only increase costs and degrade performance. Files written directly by developers have a slight improvement effect, but even that is not cost-effective.

Why Did These Results Occur?

1. Redundancy with Existing Documentation

Context files generated by LLMs mostly repeated information already in the repository, such as READMEs, docs/, and example code. When the researchers removed all documentation from the repository and then experimented, the LLM-generated context files actually showed an average performance improvement of 2.7%. This means that in repositories with sufficient documentation, context files become pure noise.

2. Repository Overviews Are Practically Meaningless

A typical recommendation for context files is to "include a repository structure overview," but the experiments showed that it did not help find relevant files faster at all. Instead, GPT-5.1 mini exhibited inefficient behavior, repeatedly reading the context file multiple times when it was present.

3. Unnecessary Requirements Make Tasks Difficult

If the context file contains instructions such as "This project uses uv" or "Run tests with pytest," the agent faithfully follows them. The problem is that this process consumes more exploration, more testing, and more inference tokens. GPT-5.2's reasoning tokens increased by 22% when a context file was present. The problem isn't that the agent ignores instructions, but that it follows them too well.

4. Stronger Models Don't Necessarily Create Better Context Files

Using context files generated by GPT-5.2 in other agents showed a slight improvement (+2%) in SWE-bench Lite, but a -3% decrease in AGENTbench. There was no consistent difference even when the prompts were changed.

Practical Implications

What You Can Do Right Now

Don't use LLM-generated context files. Automatically generating them with /init only increases costs and doesn't provide performance benefits at this point.

If you're going to write them manually, keep them to a minimum. The paper's conclusion is clear: don't include things like "repository overviews," "directory descriptions," or "architecture diagrams." Agents can already explore the repository with grep and find. Instead, only include information that the agent cannot find on its own.

Examples of effective context files:

• This project manages dependencies with uv (not pip)
• Run tests: make test-unit (do not call pytest directly)
• Only Python 3.12 is supported in CI
• Do not touch src/legacy/

Examples of ineffective context files:

• Project overview and purpose description
• List of directory structures
• Coding style guide (lint takes care of it)
• Architecture description

Effective in Repositories with Insufficient Documentation

An interesting twist is that context files are actually helpful in repositories with little documentation. If you are using agents in internal projects or repositories with poor documentation, writing down the build/test commands and core dependencies can have a significant effect.

What About Languages Other Than Python?

This paper only covered Python. Python is a language richly included in LLM training data, so the model already knows the conventions of pip, pytest, Django, Flask, etc. The effect of context files may be different for niche languages such as Rust, Zig, or Elixir. This remains a topic for future research.

Application to Our Workflow

Key takeaways from the paper:

1. Don't write lengthy repository structures in AGENTS.md — agents already have exploration capabilities
2. Only write what the agent cannot know — build tools, special commands, forbidden areas
3. Avoid automatic generation (/init) — if it overlaps with existing documentation, it's purely harmful
4. Cost monitoring is essential — token usage may increase by 20%+ after adding context files
5. Actively create them only in undocumented projects

Ultimately, a good context file is short, specific, and contains only information that the agent cannot discover on its own. The intuition that "more information = better results" does not hold true for current coding agents, at least.

Paper: Gloaguen et al., "Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?", ICML 2026. arXiv:2602.11988

Benchmark Code: github.com/eth-sri/agentbench