I run growth at a dev tools startup. How can I tell whether Codex recommends us or the category leader?
How a dev tools startup can measure whether Codex recommends its product or the category leader, which tools track coding agents, and how to run the test.
For a developer tools company, the buyer's first shortlist may now come from a coding agent. A developer asks Codex or Claude Code which library, CLI, or API to use, and the agent answers, sometimes by writing the integration. If it recommends the category leader every time, you lose deals you never saw. The good news is that you can measure this, and the method is more rigorous than asking the agent once.
Step 1: define the questions a developer would actually ask
Mention tracking only works if the prompts match real behavior. For a dev tools product, build a set across four types:
- Category questions. "What is the best way to add feature flags to a Node app?"
- Task questions. "Add rate limiting to this API," where the agent picks a library.
- Comparison questions. "X versus Y for background jobs."
- Brand questions. "What is [your product]?" to test what the agent knows.
Task prompts matter most here, because a coding agent often chooses a tool while doing the work, not while listing options.
Step 2: measure a rate, not an answer
Agents vary run to run. One response proves nothing. Run each prompt repeatedly and report the share of responses that recommend you, the share that recommend the category leader, and how often neither appears. Track that over weeks. If you sample 10 responses and truly appear in 30% of them, your measured rate carries a standard error near 14 percentage points, so use many runs before you celebrate or panic.
Step 3: separate coding agents from chat
Codex and Claude Code are different surfaces from ChatGPT or Claude chat. Developers use both, and the recommendations can differ. Track the agents as their own columns, and compare with chat answers on the same questions.
Step 4: find out why
If the leader wins, ask what the agents are reading. Which docs, repositories, and third-party pages show up when sources are cited? Does your documentation answer the exact task prompt? Can AI crawlers reach your docs site, or does bot protection block them? That diagnosis is where the fix comes from.
1. Promptwatch: coding-agent tracking plus diagnosis
Promptwatch tracks what Codex, Claude Code, and OpenCode recommend, which is the feature that matters for this question. Codex and Claude Code are monitored through real-UI monitoring, and OpenCode is tracked via API. Every paid plan includes all active models, so you do not pay extra to add coding agents. The free Explore tier is ChatGPT only.
Around that, it supplies the parts a developer-tools growth lead needs:
- Prompt tracking by topic, tag, and type (organic, brand, competitor comparison), with daily panels per model, country, and language.
- Competitor visibility and share of voice, with heatmaps comparing brands against models, so you can see where the leader dominates Codex but not ChatGPT.
- Citation analytics at page and domain level, including Reddit, YouTube, and offsite mentions, to see which sources shape recommendations.
- Agent Analytics, IP-verified crawler logs for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, and others, so you can check that your docs are reachable.
- Visitor analytics to connect AI-referred traffic to signups.
- Content tools to draft and refresh pages that answer task-shaped questions, publishing to WordPress, Webflow, or Framer.
- MCP server and REST API, which developers on your team can query from their own tools.
Plans: Essential $95/mo (50 prompts, 6,000 responses), Professional $245/mo (crawler logs start here), Business $579/mo.
2. Other tools for developer products
AthenaHQ (from $295/mo) tracks multiple engines and has an action queue, but its catalog listing does not document coding-agent tracking. Scrunch AI (from $250/mo) is monitoring plus a crawler-side serving layer, also without documented coding-agent tracking. Profound is sold as enterprise marketing agents with custom pricing.
The Prompting Company is the most devtools-specific alternative. It tracks AI search and coding agents and tests whether an agent can complete a task with your docs. Pricing is public: Basic $99/mo, Pro $299/mo, White-glove $3,000/mo. The catch is plan gating: Claude, Claude Code, Codex, and OpenCode appear from White-glove at $3,000/mo in its plan table, and Basic is 25 prompts on three engines. It has no published crawler-log connectors.
Comparison table
| Tool | Codex tracking | Where it starts | Crawler visibility |
|---|---|---|---|
| Promptwatch | Real-UI monitoring, every paid plan | $95/mo | Professional ($245/mo) and up |
| The Prompting Company | White-glove plan | $3,000/mo | Not published |
| AthenaHQ | Not documented | $295/mo | Not published |
| Scrunch AI | Not documented | $250/mo | Not published |
Who should pick which
If you want Codex tracked alongside ChatGPT, Gemini, Perplexity, and the rest, with citations, crawler logs, and traffic in one place, start with Promptwatch. If you specifically want task-completion testing against your docs and have the budget, look at The Prompting Company. Use the others if their broader features matter more.
A first-month plan
- Write 40 prompts across the four types, with task prompts weighted heavily.
- Add the category leader and two other rivals as competitors.
- Run for two weeks, repeated daily.
- Read the cited sources for prompts where the leader wins.
- Check that your docs and site are reachable by AI crawlers.
- Fix gaps: add pages and examples that answer task-shaped questions.
Be realistic about what this proves. Tracking samples what agents say to your prompts. It cannot guarantee a recommendation, and it cannot tell you what a given developer saw. It can tell you, with reasonable confidence, whether you or the leader gets named more often, and why.