Ad MCP Benchmark 2026: Evidence, Reliability, and What We Can Actually Compare

Executive summary

Advertising MCP now includes official servers from Google, Meta, TikTok,

Amazon, Microsoft, and Snap, plus independent products such as Pipeboard,

AdsMCP, AdKit, Adspirer, and PaidSync. They differ in platform breadth,

read/write scope, safety controls, architecture, and auditability.

This report does not turn those differences into an overall reliability

ranking. We found no public, workload-matched production outcome data for the

other providers. A small synthetic sample would not fix that evidence gap.

Instead, this edition publishes three bounded forms of evidence:

• a primary-source capability and safety review;
• a reproducible task-level benchmark harness and prespecified methodology;
• Synter's aggregate production terminal outcomes plus 100-repetition

protocol controls against two official open-source MCP servers.

The central rule is simple: **transport success, terminal success, and semantic

success are different measurements**.

• Transport success means a connection/session remained usable.
• Terminal success means a tool returned without MCP isError or an audited

terminal failure.

• Semantic success means a task-specific hidden verifier confirmed the

intended result and, for writes, independent post-state.

Synter's production logs measure terminal outcomes, not semantic correctness.

The protocol controls measure initialization and discovery, not advertising

work. Neither is a head-to-head product score.

Provider evidence at a glance

Tool counts are architecture and breadth facts, not quality scores. A

dispatcher with three MCP tools can route hundreds of capabilities; a native

catalog exposes each operation separately. Platform count also says nothing

about depth.

Observed Synter production terminal outcomes

The audit ran at 2026-09-03T02:49:35.309Z in a REPEATABLE READ, read-only

database transaction. It includes active customer workspaces and excludes

internal, sandbox, developer-sandbox, CI/test, inactive, and no-active-member

organizations.

“Substantive” excludes only credits, whoami, get_credit_balance, and

get_billing_status. Agent tool executions use a separate table and

denominator and are not mixed into MCP results. The earliest trustworthy

customer MCP event is July 21, 2026 at 12:27:58 UTC.

These are audited terminal outcomes. They do not establish that an ad report

contained the right numbers, that a campaign mutation reached the intended

state, or that a business objective was achieved. The semantic-success rate is

therefore not measured, not 96.28% or 81.32%.

Distribution across customer workspaces

The distribution prevents a high-volume aggregate from hiding the lower tail.

It is descriptive, not a provider comparison.

Failure families

Of 2,343 substantive failures in the trailing 30-day window:

The committed aggregate contains no customer identifiers, request/response

bodies, secrets, account IDs, or payment information.

Comparable protocol controls

We ran one frozen discovery task 100 times cold and 100 times warm against two

official open-source controls: MCP's Filesystem reference server pinned to

@modelcontextprotocol/server-filesystem@2026.8.31, and Microsoft Playwright

MCP pinned to @playwright/mcp@0.0.80.

These controls test stdio startup, MCP initialization, tools/list, pagination,

schema validation, and catalog hashing. A warm discovery run reissues

tools/list on an initialized session; it is not an in-process cache lookup.

No model, credentials, ad account, or tool execution is involved.

All four cells used n=100. A perfect 100/100 proportion has a 95% Wilson

interval of 96.30%–100%. “Semantic” here means only that discovery returned at

least one valid tool; terminal success is not applicable because no tool was

called. These results validate harness interoperability and offer reproducible

protocol baselines. They do not rank advertising MCP reliability.

Preliminary semantic harness validation

The deterministic fixture run executed 90 attempts: nine tasks × five

repetitions × cold and warm modes. It used no model or external service.

• 80/90 attempts met the semantic assertion.
• Expected approval/authentication refusals were semantically correct while

terminal success was false.

• 10 injected read timeouts recovered after one retry.
• 10 unknown-outcome mutation timeouts were not retried; post-state found

exactly one mutation each, but semantic success remained false because no

terminal task result returned.

• The publication gate remained false because n=5, no Azure-routed model was

used, and the contender was a fixture.

A separate 20-attempt Filesystem integration fixture completed discovery and

independently verified writes in every attempt. At n=5 per cell, it is an

interoperability check, not comparative evidence.

Methodology

The publication corpus is frozen before scored execution. Tasks cover

read-only discovery, fixed-window analytics, generated artifacts,

approval-required refusals, safe draft/paused mutations, authentication,

schema discovery, retry, timeout, idempotency, and unknown-outcome recovery.

Only matched capability cells are compared.

For every scored cell we freeze the Azure-routed model deployment, agent loop,

system/task prompt hashes, temperature and seed behavior, token and tool-call

budgets, MCP revision, contender revision, live schema hash, transport, account

permissions, API version, quota tier, environment image, timeout, retry policy,

fixture, and independent verifier revision.

The publication minimum is 100 independent repetitions per task, contender,

and cold/warm cell. Binary rates include two-sided 95% Wilson intervals.

Latency reports n, p50, p95, and p99, with a deterministic 2,000-resample

bootstrap interval for p95 when n ≥ 20. Cold and warm latency are never pooled.

The harness publishes first-attempt and recovered semantic success separately,

records every retry, and never automatically retries a non-idempotent mutation

after an unknown outcome. A cell qualifies only when n ≥ 100, the semantic

Wilson lower bound is at least 95%, timeout rate is at most 1%, required

post-state checks pass, and no retry-policy or secret-scan violation occurs.

There is no unweighted global leaderboard. Missing evidence is reported as

“not published,” not zero. Official status, source availability, stars, test

count, or a successful protocol handshake are not proxies for semantic

reliability.

The official MCP conformance suite

and MCP Inspector cover

protocol behavior and debugging, not advertising-task correctness.

Accenture MCP-Bench evaluates generic

end-to-end tool use; its scores are not imported here.

Evidence-backed findings — not a reliability ranking

• Official ad MCP availability is broader than open-source availability. Meta,

TikTok, Amazon, Microsoft, and Snap have official hosted services, but no

public vendor-owned implementation source was verified for them.

• Google Ads is an official, source-available, deliberately read-only baseline.

Microsoft Advertising and Snap are also documented as read-only hosted

services.

• Meta, TikTok, and Amazon document write workflows, but public documentation

does not provide workload-comparable production outcome distributions.

• Independent providers expose materially different architectures and evidence

levels. Pipeboard and AdsMCP publish implementation source; AdKit, Adspirer,

and PaidSync document hosted runtimes whose server implementations are not

fully public.

• Synter is the only provider in this review with a committed

denominator-backed production terminal aggregate. That is useful operational

evidence, but not proof of comparative semantic superiority.

Reproduce and audit

The repository includes the Python harness, frozen corpora, contender catalog,

balanced schedule generator, JSONL/JSON/CSV/Markdown reports, CI workflow,

primary-source inventory, methodology, and adversarial bias review.

cd benchmarks/mcp-reliability
export PYTHONPATH="$PWD/src"
python -m unittest discover -s tests -v
python -m mcpbench.cli run \
  --corpus config/corpus.json \
  --contender config/contenders/fixture.json \
  --output /tmp/mcp-reliability \
  --repetitions 5 \
  --modes cold,warm
python -m mcpbench.cli scan-secrets /tmp/mcp-reliability

The machine-readable production aggregate is

results/production-terminal-2026-09-03/aggregate.json; protocol-control raw

attempts and aggregates are under results/protocol-*-2026-09-03/.

Limitations and next evidence required

No live Synter-versus-provider advertising task result is claimed. A publishable

comparison still requires dedicated test accounts, matched permissions and

quota, immutable provider pins, archived live schemas, one frozen Azure-routed

agent, runner-enforced matched balanced scheduling, hash-bound post-generation

secret-scan finalization, independent provider readback, and 100 repetitions per

matched cell.

The source review is dated September 3, 2026. Hosted services can change without

a public commit. Capability claims are labelled source-verified, officially

documented, or vendor-documented. The authoring team performed the initial

adversarial review; an independent reviewer should inspect the frozen protocol

before any live product comparison.

This is a dated benchmark, not a permanent verdict. Updates should preserve old

results, version methodology changes, and publish every exclusion and

invalidation.