Comparing LLM API Gateways — The Traps in Markup, Billing, Regions, and Training-Data Use

An “LLM gateway” that bundles APIs from multiple vendors (OpenAI, Anthropic, Google Gemini) behind a single endpoint with a shared auth, billing, and observability layer has become a standard component when building generative-AI applications. Switch providers and your call code stays the same; failover and cost-optimized routing all live on one API surface. The poster child for that convenience is Vercel AI Gateway.

But the moment you start looking for “a Vercel AI Gateway–like experience from some other service,” you notice how strikingly different each vendor’s pitch is. A “no markup” headline can still hide a separate fee on your total bill; whether billing is prepaid or postpaid changes your production availability design; and pointing an endpoint at a region does not stop the data from leaving the country. These threads are tangled together.

This post uses Vercel AI Gateway as the reference point and compares OpenRouter, Cloudflare AI Gateway, Helicone, Portkey, and LiteLLM, along with the cloud aggregators Amazon Bedrock, Azure AI Foundry, and Vertex AI, across four axes. Prices, regions, and policies change fast, so treat this as a snapshot of primary sources as of late June 2026, and verify against each vendor’s official pages as of your own check date before adopting anything in production.

The Four Axes — Where the Differences Show Up

When choosing an LLM gateway, the axes worth comparing come down to four.

Token markup and fees: is there a surcharge on the upstream provider’s per-token price? Even if there isn’t, is a credit-purchase fee or per-feature charge stacked on separately?
Billing model: is it prepaid (you buy credits up front) or postpaid (subscription or usage-based, invoiced after the fact)?
Region and data residency: where does inference physically run, and where do logs and caches live? Can you keep it inside Japan or APAC?
Training-data use and logging: can you guarantee prompts and outputs are not used to retrain models? Does the gateway itself store the log body?

The first axis is the one most often misunderstood. To put the conclusion up front: none of the services surveyed adds a direct markup to the upstream per-token price itself. But “no markup” and “no extra cost on the total” are entirely different claims, and nearly every service recovers a fee somewhere. That trap is the lens for the rest of this post.

The Main Services, Side by Side

First, the whole picture on one page. Legend: ◎ = clearly favorable / supported, ○ = conditional, △ = limited / caution, ✕ = not applicable.

Service	Token markup	Payment / transaction fee	Billing	Japan / APAC region	Trains on your data
Vercel AI Gateway	None (0% even on BYOK)	Payment-processing pass-through (~3% measured); team-wide ZDR $0.10 / 1,000 req	Prepaid (credits)	△ no JP endpoint published	✕ No (default deletion + ZDR)
OpenRouter	None (old 10% dropped)	5.5% card ($0.80 min) / 5% crypto; BYOK 5% above 1M req/mo	Prepaid (invoicing on Ent.)	△ EU yes, no Japan	✕ No (no logging by default)
Cloudflare AI Gateway	None	5% on Unified Billing credits	Prepaid / BYOK postpaid	△ Tokyo edge, not pinnable	✕ No (ZDR on Unified Billing)
Helicone	None	None (standard Stripe only)	Postpaid (subscription)	○ US / EU selectable	△ no explicit no-train claim
Portkey	None (BYOK)	None (subscription + log overage)	Postpaid (subscription)	○ Japan via self-host	✕ No
LiteLLM	None (OSS self-host)	None (pay providers directly)	Self-host / Ent. quote	◎ deploy anywhere	✕ No (no external storage)
AWS Bedrock	None (own rates)	None	Postpaid (AWS invoice)	◎ Tokyo + Osaka	✕ No (contractual)
Azure AI Foundry	None (own rates)	None	Postpaid (Azure invoice)	○ Japan East (Standard)	✕ No (default)
Vertex AI	None for Gemini, premium on OSS	None	Postpaid (GCP invoice)	◎ Tokyo, residency guarantees	✕ No (not without permission)

Now to each axis in turn.

Markup and Fees — Zero Markup Is Not Free

The thing practitioners most often misread is pricing structure. It all comes down to one point: “no markup on the model price” and “no extra cost on the total” are different.

Cloudflare AI Gateway, when you use its Unified Billing feature that removes the need to manage per-provider API keys, passes the upstream inference price through at cost but adds a flat 5% fee on the credits you purchase. Buy $100 of credits and you are charged $105. The core gateway features themselves are free, with no per-request fixed fee.

OpenRouter dropped its old per-token markup of roughly 10% and now headlines “no markup on model prices.” Instead it charges a credit-purchase fee of 5.5% (with an $0.80 minimum) on card and 5% on crypto. Because that $0.80 minimum bites, the effective rate on small top-ups under $20 can spike to anywhere from 8% to 16%. BYOK (bring your own key) is free for the first 1M requests a month; above that, a routing fee equal to 5% of what the same model and provider would normally cost applies. Note also third-party reports of high effective markups on specific direct routes (one analysis cited 100% on a Claude model), so if you burn a lot of Anthropic tokens, verify per model and consider moving to BYOK.

Vercel AI Gateway states a token markup of exactly 0%, BYOK included. But its docs note plainly that “if you bring your own key we add no markup, but you are responsible for payment-processing fees,” and one review measured roughly $3.20 of fees on a $100 credit purchase. On top of that, the governance features carry separate per-request charges. Team-wide Zero Data Retention (ZDR) enforcement costs $0.10 per 1,000 successful requests, the team-wide provider allowlist the same, and custom reporting runs $0.075 per 1,000 writes and $5 per 1,000 queries. Per-request ZDR, however, is free, so in production you can dodge that surcharge by using the per-request header instead of team-wide enforcement.

Helicone and Portkey are designed around BYOK, so you pay the upstream for tokens directly and the gateway charges a subscription for monitoring and log storage. Helicone tiers are Hobby (free, 10K requests/mo), Pro $79/mo + usage, Team $799/mo + usage, and Enterprise, with 0% markup front and center. Portkey is Free (dev, 10K logs/mo), Production $49/mo (100K logs/mo, +$9 per additional 100K up to 3M), and Enterprise. LiteLLM, so long as you self-host the OSS edition, incurs no broker fee at all and pays the upstream provider directly.

In procurement terms, this splits into three molds: Cloudflare and OpenRouter are “easy to pay, slightly thicker fee structure”; Vercel and Helicone are “easy to pay, upstream-at-cost emphasized”; and Portkey and LiteLLM are “billing leans toward SaaS, contract, or self-host, but operational control is strong.” If you want multiple providers on one invoice, Cloudflare and OpenRouter are the clear picks; if you face strict cost accountability, Vercel and Helicone are easier to explain internally.

Billing Model — Prepaid vs Postpaid Changes Your Availability Design

Billing splits into prepaid (you load a deposit first) and postpaid (subscription or usage-based, invoiced afterward), and this maps directly onto production availability design.

Prepaid is exemplified by Vercel, Cloudflare (when using Unified Billing), and OpenRouter. On Vercel AI Gateway’s paid tier you buy AI Gateway Credits up front and they are deducted per request. When the balance hits zero the gateway immediately returns HTTP 429 (Too Many Requests) and the integration breaks, so auto top-up (recharging automatically when the balance drops below a threshold) is effectively mandatory in production. OpenRouter’s pay-as-you-go works the same way; purchased credits have an expiry (a third party reports directly purchased credits lapse after about 12 months of inactivity), and enterprise customers above ~50M requests/month can, after review, open invoicing via purchase order or a credit line.

By contrast, Helicone and Portkey are postpaid. Token consumption itself is billed directly by the provider later via BYOK, and the gateway charges a subscription for its logging and monitoring. On Portkey’s Production tier ($49/mo) you get 100K logs included, with overage tallied at the next invoice at $9 per 100K logs. This shape carries very little risk of the whole system halting on a gateway balance running dry during a usage spike, which is an advantage for production availability. The cloud aggregators (Bedrock, Azure, Vertex AI) are postpaid as well, with billing folded into each cloud account.

If budget control is your top priority and you never want service to stop during a burst, a postpaid subscription/usage hybrid (Helicone, Portkey) is the safe bet. If you would rather pay at cost for exactly what you use and keep internal cost accounting simple, prepaid suits you.

Region and Data Residency — “Tokyo” Is Not “In-Country Processing”

When you put generative AI into a regulated industry, where the gateway routes traffic and where it leaves logs and caches becomes a serious question. And this is where the biggest trap lives: “setting the region to Tokyo” and “the data staying inside Japan” are entirely different claims.

On this axis the SaaS routing gateways are almost a clean sweep of misses. Vercel AI Gateway is a single global endpoint (https://ai-gateway.vercel.sh/v1) and publishes no Japan regional endpoint or APPI-style residency guarantee; its data-control story is ZDR (immediate deletion), not in-country processing. Cloudflare AI Gateway runs on a global edge including Tokyo, but the execution location is not guaranteed and inference cannot be pinned to Japan. A Cloudflare community answer further notes that AI Gateway is incompatible with Regional Services and Geo Key Manager, and that a customer-metadata boundary only covers log metadata. OpenRouter offers an EU-only base URL (https://eu.openrouter.ai) “Sovereign AI” for enterprise, but no Japan in-region option is published.

The SaaS exceptions are Helicone and Portkey. Helicone lets you explicitly choose US or EU in its cloud; pick EU and pointing traffic at eu.api.helicone.ai keeps data and logs processed and stored only within the EU. You can also force a provider’s regional lock with a parameter like model: "gpt-4o/azure/eu-frankfurt-deployment", pinning routing without rewriting code. Portkey’s managed offering provides US, EU, and India, with no off-the-shelf Japan/APAC region today. Its enterprise VPC/hybrid/air-gapped deployment changes the calculus, though. The gateway core ships as an OSS Helm chart you can deploy as proxy nodes inside your own AWS, GKE, or Azure VPC, with the log store bound via the LOG_STORE environment variable to your own GCS or S3 bucket, keeping 100% of the traffic records inside your own security boundary. LiteLLM, being self-hosted by design, lets you place the deployment region entirely at will in the same spirit.

And the option that most cleanly delivers “data genuinely processed inside Japan” is not a SaaS router but a cloud aggregator. I covered this in detail in a separate post, Surveying Services That Serve OSS Model APIs in a Japan Region, but AWS Bedrock in Tokyo (ap-northeast-1) and Osaka, Azure OpenAI in Japan East (Standard deployment), and Vertex AI in Tokyo (asia-northeast1) are the few contractually backed realistic answers. Caveats apply, though. Google Cloud’s docs state that a regional endpoint alone does not guarantee data residency or in-region ML processing and that the global endpoint does not satisfy the requirement, and Azure notes that Global deployments may be processed in any of the deployed regions. Azure has no Japan Data Zone, so in-country processing requires explicitly choosing a Japan East single-region (Standard) deployment. Bedrock’s “Japan (JP) geo” profile that confines processing between Tokyo and Osaka is currently Anthropic Claude–only.

Training-Data Use and Logging — ZDR, DLP, and PII Masking

Two things: that your sensitive prompts and outputs are not used to retrain a provider’s models, and that the gateway itself holds no log body. How each service guarantees them reflects a different philosophy.

The clearest ZDR (Zero Data Retention) story belongs to Vercel and OpenRouter. Vercel’s ZDR force-routes only to endpoints whose providers have signed data-retention agreements, and the gateway itself permanently deletes prompts and outputs after a request completes by default. It also offers a “Disallow Prompt Training” control, with ZDR acting as a superset of training opt-out. OpenRouter states “We do not train on your data.” on its pricing page and does not log prompt or completion bodies by default (opting in to logging returns a 1% token discount, an unusual design). It also has a toggle that excludes auto-routing to providers that train, preventing an opaque provider from being chosen by the router.

For keeping the log body out of the gateway, Helicone and Cloudflare offer opt-out designs. Helicone’s “Omit Logs” excludes all bodies from records, and it can extract only audit metadata (token counts, latency, TTFT) asynchronously while wiping the prompt body from memory the instant it passes through. Cloudflare stores the full payload by default but can switch to metadata-only via the cf-aig-collect-log-payload: false header or gateway settings. Cloudflare also wires a DLP (Data Loss Prevention) engine into the edge that detects requests containing credit-card numbers or PII in real time, letting you choose between flagging or blocking with HTTP 400. Note the trade-off: extend DLP evaluation to the response and it must buffer the streaming reply before evaluating, badly eroding the TTFT benefit, so if you need low latency, scan input only.

Portkey takes a different approach, implementing native PII detection and redaction on Production and Enterprise. An input like “my phone number is (555) 123-4567” is transformed into a placeholder such as [PHONE_NUMBER_1] before being sent upstream, so the provider never touches the personal-data body and can still return an inference. Log retention is finely controlled too: 3 days on Free, 30 days on Production, custom on Enterprise.

The cloud aggregators are the strongest on the contractual side. The Amazon Bedrock FAQ states verbatim that “AWS and the third-party model providers will not use any inputs to or outputs from Bedrock to train Amazon or any third-party models,” and it does not store prompts in standard operation. Microsoft likewise commits that Azure OpenAI does not use prompts, outputs, embeddings, or fine-tuning data to train foundation models without your permission. There is, however, a default 30-day abuse-monitoring retention, and disabling it via Modified Abuse Monitoring / ZDR is gated behind Microsoft approval and an EA/MCA contract. Google states customer data is not used for training without permission and lets the customer control where it is stored.

One organizational note on Portkey: on April 30, 2026 Palo Alto Networks announced its acquisition, closing May 29 and folding it into Prisma AIRS. The gateway was also open-sourced as Gateway 2.0, so roadmap and pricing may shift through the transition; verify directly before adopting.

The Shared Architecture

The services here share a skeleton despite their differences: a gateway layer that bundles a unified API, routing, auth, and observability sits in front of the application, with multiple providers hanging off the back. The difference is whether that gateway centers on managed credits, on BYOK, or on self-hosting.

flowchart LR
    A["App / Agent"] --> B["Unified Gateway API"]
    H["Managed Credits"] --> B
    I["BYOK / Provider Keys"] --> B
    B --> C{"Routing policy"}
    C -->|"Cost / latency"| D["Provider A"]
    C -->|"Failover"| E["Provider B"]
    C -->|"ZDR / region"| F["Provider C"]
    B --> G["Observability / Logs / Spend"]

Vercel, Cloudflare, OpenRouter, and Helicone keep managed billing or credits relatively front and center, while Portkey and LiteLLM weight more toward BYOK and self-hosting. Lining up only the surcharges spelled out in public materials, you get Cloudflare 5%, OpenRouter 5.5%, and Vercel and Helicone 0% (though Vercel may incur payment-processing fees).

Other Options — Martian, Unify, Braintrust

Beyond the main six, a few routers can fit depending on the use case. Martian is a model router that predictively routes to the cheapest adequate model, reported as pass-through plus roughly a 5.5% fee (third-party-reported, so verify). Unify is a neural router optimizing cost, latency, and quality with BYOK virtual keys, but pricing for its LLM-routing product is not published (and a different company by the same name exists, so don’t conflate them). Braintrust bundles an AI proxy into an eval/observability platform; the proxy itself is free with no token markup, while the platform is a postpaid subscription at $249/mo on Pro. None publishes a Japan/APAC region, so if data residency is a requirement, an enterprise VPC or on-prem deployment is the premise.

Choosing by Requirement

Here is the whole comparison distilled into a selection strategy worked backward from requirements.

Add managed multi-LLM access with the least friction: Vercel AI Gateway or Helicone
- Both are OpenAI-compatible and light to adopt, headlining 0% markup. On Vercel, prefer free per-request ZDR over team-wide enforcement in production to avoid the surcharge.
Prioritize catalog breadth, a privacy filter, and EU routing: OpenRouter
- Its strengths are 400+ models, a “exclude training providers” toggle, and an EU-only URL. Watch the fee spike on small top-ups, and verify Claude-route effective markup per model.
Already on the Cloudflare stack: Cloudflare AI Gateway
- Observability, caching, fallback, custom providers, and DLP integrate on one operational surface. The deciding factor is how you weigh the 5% Unified Billing fee.
Top priority is VPC, isolation, governance, and enterprise control: Portkey or LiteLLM
- Portkey can satisfy a Japan region and PII masking via self-host. LiteLLM maximizes both placement and data control through OSS self-hosting.
Need in-country Japan processing backed by contract: AWS Bedrock Tokyo / Azure OpenAI Japan East / Vertex AI Tokyo
- SaaS routers fundamentally cannot meet APPI-style in-country processing. The realistic answers are Bedrock Tokyo (Claude via the JP geo), Azure’s Japan East Standard deployment, and Vertex with the regional endpoint stated explicitly.

The deciding condition is simple. If you must back in-country Japan processing with a contract, the answer is a cloud aggregator or a Portkey/LiteLLM self-host, not a SaaS router. If in-country processing is not a requirement, the easy paths are the low-friction, 0%-markup Vercel and Helicone, or OpenRouter for catalog and policy control.

Wrap-Up

The key points:

None of the surveyed services adds a direct markup to the upstream per-token price, but “no markup” and “no extra cost on the total” are different, and nearly every vendor recovers a fee somewhere.
The spelled-out surcharges are Cloudflare 5%, OpenRouter 5.5% ($0.80 min), and Vercel and Helicone 0% (Vercel may incur payment-processing fees). Vercel stacks per-feature charges like team-wide ZDR.
Vercel, Cloudflare, and OpenRouter are prepaid and halt when the balance runs out; Helicone, Portkey, and the cloud aggregators are postpaid with higher burst tolerance.
“Region = Tokyo” and “data stays in Japan” are different problems. SaaS routers are fundamentally unsuited to in-country processing; for APPI, the realistic answers are Bedrock/Azure/Vertex or a Portkey/LiteLLM self-host.
The most direct no-train statement is OpenRouter’s; Vercel pairs default deletion with ZDR; the cloud aggregators are strongest contractually. Helicone and Cloudflare can keep the log body out; Portkey offers PII masking.

To stress the point one more time: prices, regions, and policies move fast in this space, and every vendor’s pages get rewritten over short windows. Treat this as a map as of late June 2026, and always check primary sources as of your own date before adopting in production.

That’s a comparison of LLM gateways in the Vercel AI Gateway mold across markup, billing model, region, and training-data use, reported from the field.