TL;DR
- DX problems are defects. They have reproductions, severity, and regression tests, and they should be tracked that way instead of filed as polish.
- The fastest way to find them is to stop reading your own code and walk your own first-run path with fresh credentials.
- Coding agents are now a primary consumer of developer surfaces, and they fail differently than people. A person who hits a confusing error opens the docs. An agent invents a plausible wrong answer and proceeds with confidence.
- We audited our own developer surface and found four defect classes worth naming: duplicate namespaces, errors that restate themselves, docs pointing at a host that stopped answering, and undocumented equivalence.
- The durable fix is not better docs. It is making documentation drift fail a test.
DX Is Not Polish
Most teams file developer experience under design. It gets a nice landing page, a syntax-highlighted code sample, and a quarterly cleanup. Then it gets deprioritized, because nothing is visibly broken.
That framing is wrong, and it is expensive. A developer who copies your quickstart and receives a 400 has hit a defect with a clean reproduction. The severity is high, because it lands at the exact moment a person is deciding whether your product works. The blast radius is every new account. If a payment endpoint failed for every first-time caller you would page someone. When a quickstart does it, most teams call it a docs task.
Treat it as a bug class and everything downstream changes. Bugs get reproductions, owners, and regression tests. Polish gets a backlog.
The Test: Walk the Path, Do Not Read the Code
The reason these defects survive is that the people who could find them cannot see them. If you built the API, you have credentials in your shell, the right host memorized, and the argument names in your fingers. You read the code and conclude it works, because you are reading the implementation, not experiencing the product.
So do not read. Walk. Take the published quickstart, on a clean machine, with a key you just minted, and run each command exactly as written. Change nothing. Fix nothing silently. Every place you reach for knowledge that is not on the page is a defect, and the instinct to patch it from memory is the instinct to suppress.
We did this against our own production surface. Here is what it turned up.
Four Things We Found
1. One concept, three names
A campaign read existed under three different identifiers: one for the agent tool surface, a different one for the REST parameter, and a third for the file on disk. All three were correct in their own context. None of them worked in the other two.
Our own quickstart shipped the agent tool name inside the REST example. The run-script reference now names the dispatchable identifier, and the MCP tool list names the tool-surface one. The single command a new developer copies first returned an unknown tool error. The same trap had already shipped inside our SDK, where an internal filename leaked into the published package and broke two platforms for anyone using the convenience method.
The lesson is not "be careful with names." It is that duplicate namespaces for one concept are a defect in the design, and every surface that touches them will eventually pick the wrong one. If you cannot collapse them, you owe the developer an explicit table showing all of them side by side, plus a test that fails when a sample uses the wrong one.
2. The error message that restated the error
Our REST surface returned this to a bad key:
{
"error": "Invalid or expired API key",
"help": "Check that your API key is correct and not revoked"
}Read the help field again. It restates the error and adds nothing. It does not say where to get a key, how to tell a revoked key from a wrong-environment key from one truncated on paste, or which headers are accepted.
Meanwhile our agent surface, same product, returned a numbered "TO FIX" block naming the likely causes and linking to the portal. Two front doors to one system, two completely different experiences, and nobody noticed because nobody walks in the front door twice.
An error message is documentation that arrives at the exact moment it is needed. It is the paragraph that does the most work in your entire developer surface, and it is usually the one nobody reviews. Ours now name what happened, the likely causes, and where to go next.
3. Docs pointing at a host that stopped answering
A published example referenced a hostname that no longer resolved to a running service. Not a 500. Not a helpful redirect. No answer at all.
This one is worth dwelling on because it is invisible to every check most teams run. The docs built. The links were not broken, because the hostname was inside a fenced code block, and link checkers do not execute code samples. CI was green. The only way to catch it is to run the sample, which is exactly what nobody does after the first week.
Code samples are untested code shipped to production. They deserve the same treatment as any other untested code.
4. Undocumented equivalence
Our API accepts a key in any of three headers. All three work identically, and none is preferred. The authentication reference now says so in one sentence.
Our docs named one. So developers, and agents, spent real effort deciding which was correct, treating the omission as a signal. One of our own evaluation criteria had to be rewritten because it encoded a preference that does not exist.
When your system genuinely does not care, say so out loud. Silence reads as significance. "All three are accepted" is one sentence, and it retires an entire category of question.
Agents Fail Differently Than People
This is the part most DX advice has not caught up to.
A person who hits an ambiguous error stops and reads. An agent does not stop. It produces a confident, plausible, wrong next call, and it keeps going. Ambiguity that costs a human thirty seconds costs an agent an entire run, and the agent will not tell you it was confused, because it was not. It was sure.
We watched this happen in a way worth reporting honestly. We put a correction in our own guidance document, written as "do not use this value." Agents reading it copied the wrong value out of the warning. We removed the wrong value and named a different one to avoid. They copied that one. The only version that worked was the one containing no incorrect literals at all.
Two things follow, and both change how you write.
Do not document the wrong answer, even to warn against it. Any literal on the page is a candidate for copying. State only what is correct.
Ambiguity is more expensive than it used to be. The cost used to be a confused person who recovers in a minute. Now it is a silent, confident failure that burns a full run and reports success. Precision that was once a nicety is now load-bearing.
Make Drift a Test Failure
Every defect above shares one root cause. Two sources of truth were allowed to disagree, and only a human reading both carefully would ever notice.
Documentation will drift. Writing it more carefully does not stop drift, it just delays it until the author moves teams. The only durable fix is to make disagreement fail a build.
What that looks like in practice:
- Generate the list, do not maintain it. Any table a person hand-maintains alongside a real registry will drift. One of our surfaces never had this problem, purely because it was generated from the source rather than typed next to it.
- Diff every published sample against the real registry. We now sweep every code sample in our docs and assert each identifier actually exists. That sweep is what found the quickstart defect, and it was the only one left.
- Test the argument surface, not just the endpoint. A call can reach a live service and still be wrong in five flag names. Our SDK now has a test that drives every dispatch path and diffs the flags each one emits against the arguments the backend scripts actually accept.
These are unglamorous tests. They are also the only reason the same defect does not come back in six weeks.
Hand the Page to the Agent
One more thing changed how our docs get consumed. Every page on our documentation site is now available as clean Markdown by appending .md to its URL, with the navigation, sidebar, and chrome stripped out.
The reason is simple. A developer's workflow is increasingly "give this page to my agent." If your only output is rendered HTML, the agent receives your cookie banner and your mobile menu alongside your parameter table. Serving the content by itself is a small piece of work that respects how people actually build now.
On the documentation site this has a useful property. The Markdown is rendered from the same page a person reads, so it cannot disagree with the documentation. A hand-written Markdown twin would be a second source of truth, and by now you know what happens to those.
What Good Developer Experience Looks Like
Good DX is not a clean landing page. It is a shorter distance between a developer's intent and a working call, measured honestly, with fresh credentials, on the actual path a stranger takes.
For AI agent integration specifically, that distance is the whole product. An agent has no patience, no intuition, and no support ticket. It gets one read of your page and one attempt at your endpoint.
Three questions worth asking about your own surface this week:
- When did someone last run your quickstart exactly as written, changing nothing?
- If a caller sends a bad key, does your error tell them what to do next, or does it restate itself?
- If your docs and your code disagreed tomorrow, what would fail?
If the answer to the third question is "a person would eventually notice," that is the work.