AI Security & Governance

Dragos Planted 2M Canaries. Your ZDR Flag Didn't Hold.

Rushil ShahRushil Shah
13 min read
Share

Dragos seeded roughly two million prompts with one-off canary credentials and found three breached inference providers, resellers reading buyers' prompts, and ZDR-tagged requests still being stored. Here is how to audit your own routes.

TL;DR

Zero Data Retention is a contractual promise that binds, at best, the first hop of your request. Dragos seeded about two million prompts with one-off canary credentials over five months and found three breached inference providers, marketplace resellers reading buyers' prompts through ordinary request logging, and no-retention requests whose embedded links were fetched days later anyway. OpenRouter responded by off-boarding three providers and stripping 125 endpoints of ZDR eligibility. If you run multi-provider routing in production, the action item is not panic — it is route mapping, per-route canaries, and a rotation rule for every secret that has ever entered model context.

ZDR is a contract term, not an enforced control

The single most useful sentence in HD Moore's write-up is the one most coverage skipped: a privacy flag only binds the first hop, and agents will happily transmit anything they can read. Everything downstream of that first hop is governed by a chain of contracts you never signed, enforced by logging configurations you cannot inspect.

Over five months from July through November 2026, the Dragos research team submitted roughly two million prompts containing unique URLs and canary credentials to widely used AI services and watched what touched them afterwards. Three categories of finding came back: evidence of compromise at three providers, resellers on an inference marketplace reading customer prompts through standard logging, and privacy settings that were simply not honoured.

~2M prompts seeded with unique canary URLs and credentials, July–November 2026 Source: Dragos, 2026
3 inference providers with evidence of attacker access; none knew until notified Source: Dragos, 2026
125 endpoints removed from Zero Data Retention eligibility by OpenRouter Source: OpenRouter, 2026
~2/3 of “no data collection” or ZDR requests to one lab's models had their secret URLs fetched Source: Dragos, 2026

The detail that should worry anyone running agents is the mechanics of the largest breach. A couple of hundred AWS keys were each sent in exactly one request to one provider during a few short windows in late October, and every request sent during those windows was stolen — including requests the provider rejected before a model ever ran. That places the attacker at the front door, copying traffic on arrival, where retention policy is irrelevant because nothing has been retained yet. In two of the three breaches, some of the stolen requests carried the ZDR flag.

A privacy flag doesn't help when the attacker is already inside.

— HD Moore, Chief Research Officer, Dragos

Every routing hop is a party that can read the whole prompt

Routers sell a single endpoint, unified billing and automatic failover. What you are actually buying is a variable-length chain in which each link sees plaintext in and plaintext out. Dragos traced about two thousand routes; several hundred included a system they could not identify, a handful forwarded a seeded image to a company that was not on the advertised path at all, and one marketplace seller was quietly appending roughly a thousand hidden tokens to every prompt.

The reseller finding required no exploit. A seller lists a popular model at the lowest price, enables request logging, and waits for the router's price-sorted selector to hand over traffic. In one case two prompts sent through a single seller's offer had their secret URLs opened from a home broadband connection a day and a half later, followed by manual username guessing at the login page; identical prompts to the same model through a different seller were never touched. The leak follows the seller, not the marketplace.

Then there is the undisclosed hop. One key that was abused had not gone through the marketplace at all — it had gone to a different router that was quietly forwarding part of its traffic there. If you have built provider allowlists on the assumption that the vendor you contracted with is the vendor that executes your request, that assumption is the thing this research breaks.

The retention failures are a separate problem from the breaches, and arguably the more structural one. A Chinese AI lab's systems fetched the secret URLs from more than half of the prompts sent to its models between July and October, typically two or three days after the request, including about two-thirds of requests marked no-collection or ZDR. Several U.S. hosting providers resell that lab's models, and prompts sent to those U.S. providers triggered the same scraper. Buying a U.S. endpoint to keep data in the U.S. did not keep data in the U.S. In the strangest case, a seeded link surfaced in a Feishu chat 24 days after the prompt was sent, opened within two minutes from three desktops and three phones — read from a stored copy of a prompt that had been submitted to a U.S.-based provider with the ZDR flag set.

How to map where your prompts actually go

The methodology is reproducible with a weekend of work and a cheap VPS, and it is the only thing that converts a vendor assertion into evidence. Dragos combined two techniques: callback assets embedded in the prompt, and deliberately malformed requests that make each hop wrap the error from the hop behind it, producing three or four nested envelopes in a single response that can be fingerprinted against a signature database.

Route-mapping an inference path you do not control

1
Enumerate the routes, not the models

Export every model ID, provider preference block and fallback entry in your orchestration layer. One model name can resolve to a dozen endpoints; ZDR status attaches to the endpoint.

↓
2
Pin the upstream and kill silent failover

Set an explicit provider allowlist and disable fallbacks for the audit window. A route that can change under load is a route you cannot attest to.

↓
3
Seed one-off canaries per route and per flag

A distinct credential and a distinct URL for each (provider × model × privacy setting) cell. Shared canaries tell you something leaked; unique ones tell you which contract was broken.

↓
4
Trace with malformed requests

Send slightly broken payloads and keep the raw error body. Nested error envelopes expose intermediaries that the route metadata never mentions.

↓
5
Run it continuously, not once

Provider pools churn weekly. OpenRouter said it is adapting the same testing approach to run on an ongoing basis — so should you, on your own traffic.

Two practical notes the research write-up does not have to deal with but your audit will. First, tool-enabled models create false positives: a model with web search or link preview will fetch your canary URL legitimately, so run the control arm with tools disabled before you call anything an incident. Second, absence of a hit is weak evidence. Phala's own report makes this point honestly — the canary tests sampled requests rather than monitoring every request, which is why the provider kept a wider precautionary exposure window than the two days of confirmed misuse.

bash
# Audit call: pin the upstream, refuse fallback, seed a one-off canary.
# Field names follow the OpenAI-compatible router convention; confirm
# them against your router's current provider-routing schema.

curl https://<router>/api/v1/chat/completions \
  -H "Authorization: Bearer $AUDIT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vendor/model-name",
    "provider": {
      "only": ["first-party-provider"],
      "allow_fallbacks": false,
      "data_collection": "deny"
    },
    "messages": [{
      "role": "user",
      "content": "Summarise: deploy key at https://a1b2c3.canary.example/repo.git user=svc_a1b2c3"
    }]
  }'

# Trace call: break the schema on purpose, keep the raw body.
# Nested error envelopes reveal hops the route metadata omits.
curl -s https://<router>/api/v1/chat/completions \
  -H "Authorization: Bearer $AUDIT_KEY" \
  -d '{"model":"vendor/model-name","messages":"not-an-array"}' | jq .

Fix it at the layer that owns the decision

Most teams try to solve this at the provider layer, by reading a policy page harder. The decisions that matter live in three different places, and each needs a different control.

Orchestration layer (n8n, LangGraph, custom gateways). This is where the route is chosen, so this is where the allowlist belongs. In n8n specifically, the risk is not the HTTP Request node you configured deliberately — it is the model dropdown on an AI Agent node that someone switched to a cheaper ID during a cost review, and the fallback model field that silently re-sends the entire context to a second vendor on a timeout. Pin provider preferences in the request body rather than in account settings, because account defaults do not travel with an exported workflow. Treat a change to a model ID as a change-controlled event, the same as a change to a database connection string. If you are standardising this across several workflows, the pattern we document under model coverage is to centralise routing in one credentialed sub-workflow rather than letting each agent pick its own endpoint.

Context layer. Agents leak secrets because secrets are readable. An MCP tool that returns a raw error body with an Authorization header, a shell tool that cats a .env, a retrieval step that indexes a config repo — each puts a long-lived credential into a payload that may be copied at a front door you do not control. Issue scoped, short-lived tokens to tools and never place a raw provider key in text the model can see. Together AI's own remediation list after its incident leads with exactly that: scoped or short-lived credentials, keyless access where supported, and credential checks on every pull request.

Procurement layer. Separate first-party endpoints from marketplace resale in the contract, and require that any change to the upstream pool is notifiable. This is also where you write the rotation rule.

!

Treating ZDR as a property of the model

Teams record “model X is ZDR” in their data map, then route to model X wherever it is cheapest. Retention behaviour belongs to the endpoint and the operator, not the weights — which is why 125 endpoints could lose ZDR eligibility in a single review while the underlying models stayed available.

Fix: record provider + endpoint + region + privacy flag as the unit in your data inventory, pin that tuple in the request, and fail closed when the pinned endpoint is unavailable rather than falling back to an unverified one.
!

Rotating only what the vendor's notice covers

Exposure windows in provider notices are bounded by what the provider can prove. Phala's confirmed misuse covered two days; its guidance covered October 21 through November 8 precisely because sampling cannot bound an attacker's access.

Fix: adopt a standing rule — any secret that has ever entered model context is rotated on a fixed cycle and immediately on any upstream notice, whether or not your traffic falls inside the stated window.

First-party endpoint or marketplace reseller: a scorecard

The useful axis is not price or latency. It is how many organisations can read the plaintext, and whether you can verify the count.

Procurement optionWho can read the full promptWhat you can actually verifyWhen it is the right call
Self-hosted weightsYouEverything, on your own telemetryRegulated or crown-jewel workloads where the hardware cost is justified; Dragos prices the realistic range from a $6,950 DGX Spark to a ~$570,000 eight-GPU B300 server
First-party lab APIOne company, one policyContract terms, region, subprocessor list, your own canariesDefault for anything carrying customer data or credentials
Hyperscaler-hosted model (Bedrock, Vertex, Azure)Cloud operator, under your existing agreementExisting cloud audit trail and IAM; model availability lags the labsTeams that already have the cloud contract and logging in place
Router with pinned upstreamsRouter plus the one provider you pinnedRoute traces, nested error envelopes, per-route canariesMulti-model evaluation and failover, once fallbacks are explicit
Open marketplace, price-sortedUnbounded — whoever won the bid, plus any hop they resell throughAlmost nothing; undisclosed hops were found by tracing, not disclosureThrowaway, non-sensitive, synthetic workloads only

Cheapest-token routing is a data-governance decision

Here is the part that gets left out of the breach coverage. Price-sorted routing is not neutral with respect to privacy risk — it actively selects for the operator most likely to keep a copy. A seller who logs your prompts has a revenue stream your honest provider does not, which means they can underbid. Dragos describes that economics plainly: list the popular model at the lowest price, route the traffic, absorb the difference, and collect the data from platform logs. Discounted capacity can also come from pooled consumer subscriptions, startup credits, or outright stolen keys.

So the sort: "price" line in an agent config is not a cost optimisation. It is an unreviewed decision to hand your plaintext to whichever party is currently most motivated to subsidise reading it. In builds where the per-request value is a few cents and the context contains customer records, that trade is indefensible on arithmetic alone.

The honest counterweight to all of this: none of it means stop using hosted inference. It means some workloads should not be routed at all yet. If your agent reads from a production database, holds a cloud credential, or handles regulated personal data, and your only cost-control mechanism is a cheap fallback model, do not ship that fallback. Run the sensitive path against one first-party endpoint, accept the higher unit price, and spend the saved risk budget somewhere else. We scope that boundary explicitly in implementation work, because it is far cheaper to draw it before the workflow exists than to retrofit it across forty live automations.

What to do before your next renewal

Mid-July 2026

Earliest of the three provider intrusions begins; the provider first attributes it to internal misconfiguration, then to a partner, before finding compromised internal credentials.

Oct 21–22

The window in which canary credentials sent to Phala's DeepSeek V4 Flash 0731 deployment were later misused; earliest observed misuse at 05:41 UTC on October 22.

Nov 8–10

Phala contains the incident at 05:35 UTC on November 8 and publishes a preliminary report on November 10 advising rotation for the full precautionary window.

Nov 19

Dragos publishes the canary study.

Nov 21–22

Together AI discloses that stored prompt samples from MiniMax M3, GLM-5.2 and Kimi K2.6 requests were downloaded after an AI coding agent committed a credential to a public repository; OpenRouter off-boards three providers, strips ZDR eligibility from 125 endpoints and notifies potentially affected customers.

Concretely, this week: export your route inventory; rotate every credential that has appeared in model context since June; pin upstreams and disable fallbacks on any workflow touching regulated data; and stand up a canary seed in one low-risk production path so you have your own telemetry the next time a disclosure lands. If you want that mapped against an existing agent estate, start here — and if you are earlier in the build, the architecture patterns matter more than the provider you pick.

Frequently Asked Questions

Does zero data retention mean my prompts are never stored?

It means the party you contracted with has agreed not to store them. Dragos found requests tagged ZDR that were still stolen, scraped and in one case shared internally weeks later, and OpenRouter subsequently removed 125 endpoints from ZDR eligibility. Treat the flag as a contractual commitment that binds the first hop, and verify everything beyond it with your own canaries and route traces.

How do I verify a ZDR claim from my LLM provider?

Embed a unique URL and a unique dummy credential in each request, vary the privacy flag across otherwise identical requests, and log every subsequent DNS lookup, HTTP fetch or credential use against the exact route that carried it. Run a control arm with model tools disabled so legitimate web-search fetches do not read as leaks. Absence of hits is weak evidence; presence is conclusive.

Was OpenRouter itself breached?

No. OpenRouter stated the findings showed no breach or unauthorised access of OpenRouter, and that the results concerned specific inference providers in its pool. It off-boarded three providers, removed 125 endpoints from ZDR eligibility as both a response and a precaution, notified potentially affected customers, and said it will require assurances and validation before restoring any endpoint's ZDR status.

Which secrets should I rotate after an inference provider incident?

Anything that was readable inside model context during the window: API keys, database passwords, deploy keys, webhook signing secrets, and tokens returned by tools whose output was pasted into a prompt. Phala advised rotating every key, token or password included in affected requests across a window wider than its confirmed misuse dates, because sampled canary evidence cannot bound an attacker's full access.

Are inference marketplaces always unsafe?

They are unsafe for sensitive payloads, because the pricing mechanism rewards the seller who monetises your prompts. Dragos found that leakage followed individual sellers rather than the platform, and recommends picking upstream providers yourself instead of accepting the default pool. For synthetic benchmarking, evaluation harnesses or non-sensitive bulk text, marketplaces remain a reasonable cost play.

How do I stop n8n agents from putting credentials into prompts?

Keep secrets in credential stores the model never reads, and make tools return sanitised results rather than raw HTTP bodies or shell output. Give each tool a scoped, short-lived token instead of a shared key, strip Authorization headers from error paths before they reach the LLM, and pin provider routing in the request body so a changed model dropdown cannot silently move the context to a new vendor.

Does provider-side secret filtering solve this?

Only partially. Filters match known credential shapes, so they catch AWS keys and bearer tokens but not source code, strategy documents, customer records or internal URLs. Dragos describes it as worth enabling but not a panacea. Use it as a backstop behind context hygiene and route control, never as the primary boundary.

AI securityLLM infrastructurezero data retentionagentic AIvendor due diligencen8nprompt privacyAI supply chain

Published

AI-assisted writing · Reviewed by the Twarx research team

Share:
Share

Research digest

AI Research Briefing

Honest insights on AI agents, Small Language Models, and local RAG. No hype. Only when we have something worth sending.

  • No hype, just measurable outcomes
  • Read by 2,400+ engineers
  • Unsubscribe anytime

Continue reading

More from Articles