The boundary
Shadow AI or your own LLM calls: which blind spot are you closing?
Two problems get discussed as one, and they do not answer to the same controls. Naming which one you are closing decides what you deploy, and where.
01
The tools your teams adopt without telling you
Someone signs up for a free assistant and pastes a client contract into it to get a summary. Nobody approved the tool, nothing appears on an asset register, and the exposure travels outbound over traffic you can inspect. That is a discovery and policy problem, answered at the network layer, and it has its own playbook.
02
The calls your applications make
API calls to OpenAI, Anthropic, Cohere, Workers AI or any other provider, triggered by your code and billed to your accounts. They carry your users' data, consume your budget and create compliance obligations. Nobody adopted them without permission and no traffic filter will surface them: they are sanctioned, they are yours, and in most organisations nobody holds the complete picture of them.
03
Why provider dashboards are not enough
Each provider shows you its own calls. OpenAI shows you OpenAI calls, Anthropic shows you Anthropic calls. None of them shows the total across providers, none applies your data protection policies to prompt content before it arrives, and none gives you a single audit trail. If a regulator asks you to demonstrate that no personal data reached a third-party model provider without a lawful basis, a collection of provider dashboards is not an answer. One control plane, sitting in front of every provider, is.
The vocabulary overlaps, the control set does not. The unsanctioned AI tools your teams adopt are a discovery problem on outbound traffic. The calls your own code makes are an instrumentation problem, and instrumentation is answered by a gateway, not by a filter.
The question teams actually ask sounds like this: how do we get real-time visibility into all AI and LLM traffic across providers, requests, latency and error rates, in one place? On Cloudflare those four dimensions sit in a single analytics view per gateway, and each of them is also available per request in the logs.
The decision
A dedicated platform, or the LLM gateway you already run?
Dedicated LLM observability platforms offer rich tracing, evaluation frameworks and fine-grained model performance metrics. They are genuinely valuable for engineers who need to understand why a model produced a particular output, how latency distributes across prompt types, or how a fine-tuned model compares with a base model. That instrumentation comes with code changes, a new vendor relationship and maintenance as the application evolves.
So the useful question is not which platform is better. It is which layer answers the question you actually have.
Which layer answers which question
An LLM gateway is a proxy between your applications and your providers: it intercepts every call, applies policy and logs the result. What it cannot do is reason about the quality of what came back.
| What you need to answer | Where it is answered | Verdict |
|---|---|---|
| How many calls left, to which provider, at what cost | One log line per request, on the proxy every provider already passes through. No application instrumentation. | Gateway |
| Which application, team or feature generated the spend | Custom metadata attached at call time, read by the logs, the spend limits and the routing rules alike. | Gateway |
| Whether a prompt carried personal data before it left | Detection profiles shared with your Cloudflare One environment, applied to request and response bodies. | Gateway |
| Evidence that a policy was applied, for an auditor | The action taken, the matched policy and the specific detections, recorded per request. | Gateway |
| What happens when the primary provider fails or the budget runs out | Fallback paths configured in the gateway rather than deployed in application code. | Gateway |
| Why latency spikes on one shape of prompt | Per-request duration on the gateway tells you it happens; span-level tracing in a dedicated platform tells you where. | Both layers |
| Whether the model output was accurate or relevant | Test sets, evaluation suites and human review, owned by the team that owns the use case. | Dedicated platform |
Where the comparison diverges
- If your priority is cost control, PII inspection, provider governance and compliance logging, the control plane you run for ZTNA and WAAP already has the primitives. You do not instrument application code, you point the SDK at a different base URL.
- If your priority is understanding model behaviour, no gateway will substitute for an evaluation platform, and this page does not pretend otherwise.
- The two coexist. A gateway meters and governs the call; an evaluation platform judges the answer. Running both is a normal end state, not a redundancy.
The question is not which LLM observability platform is better. It is whether you need a dedicated platform at all, or whether the control plane you already operate covers the use case you have today.
The control plane
What can you actually see and control on Cloudflare?
Seven levers on one proxy: what gets recorded, how long it survives, what stops costing money, what stops running away, what happens when a provider fails, what never leaves in the first place, and what happens when the budget is gone.
Request and response logging
Every prompt and every response is logged with the model, the provider, the timestamp, the request status, token usage, cost, duration and the client user agent. Per request, not in aggregate.
When DLP policies are active, the log also carries the action taken, the matched policy and profile IDs, the specific detection entries that fired, and whether the match was in the request, the response or both.
One thing the log does not carry on its own is an identity. Attribution to an application, a team or a user comes from custom metadata attached at call time, which is the same field the spend limits and the routing rules read, so it is worth deciding before an investigation needs it.
In practiceThe evidence layer. Attribution is one field at call time, not a default.
Log storage, and the default that surprises people
Each gateway has a storage limit expressed in number of logs, set by your plan and adjustable per gateway. This is the part worth reading twice: when the limit is reached, new logs stop being saved. The oldest ones are not rotated out unless automatic deletion is switched on.
So there are three deliberate settings rather than one: size the limit, enable automatic deletion so writing never stops, and export off the gateway if you need to keep more than the limit holds.
Export runs through Workers Logpush on a Workers Paid plan, with logs encrypted per log using an AES key that is itself RSA-encrypted with a public key you upload, so key management is part of the retention design rather than an afterthought.
In practiceThe failure mode is not losing history. It is silently stopping to write.
Caching, the cheapest lever there is
Identical requests return cached responses from Cloudflare's network, with no round trip to the provider, so repeated queries stop generating provider API costs. The share of responses served from cache is visible in the analytics view, and no application change is required: you route through the gateway and set the TTL, from 60 seconds to one month.
Three properties to design around. The cache key concatenates provider, model and request body, so changing model invalidates the cache. Matching is exact, not semantic. And a cache hit skips DLP scanning, which means a response cached before a policy change keeps being served until the TTL expires unless the cache is bypassed explicitly.
In practiceExact match only. A cached response is not re-inspected when the policy changes.
Rate limiting per application
Rate limiting stops one application, agent or user consuming disproportionate quota. You set limits per gateway, in requests per window, fixed or sliding, and the gateway enforces them before calls reach the provider. For agentic applications, where a runaway loop can burn a monthly budget in hours, it is the first circuit breaker.
What it does not do is bound cost: it counts requests, not tokens, so a loop carrying a large context can spend heavily while staying under the request ceiling. That is what the spend limits below are for.
In practiceIt counts requests, not tokens. Necessary, and not sufficient.
Provider switching
Dynamic routing composes a small flow in front of your providers: conditional branches on request body, headers or metadata, percentage splits for progressive rollouts, rate limit and budget limit nodes, and model nodes with retries and a fallback. You change the flow without touching application code, and roll back to a previous version instantly.
That covers three operational cases: a provider that errors or times out, a budget that is exhausted, a request quota that is reached. What routing does not decide is where the inference happens, which is a provider and model question rather than a network one.
In practiceFailure, budget and quota all have a fallback path. Geography does not.
PII detection with DLP
AI Gateway DLP scans incoming prompts and outgoing responses in real time, using the same detection profiles as Cloudflare One's DLP. Because those profiles are shared account-level objects, the ones already governing your web traffic apply here, with no proxy deployment and no TLS decryption to set up.
Two precisions matter before you promise anything internally. Detection flags or blocks; it does not mask or substitute, so a prompt is either allowed through or refused, never sanitised in place. And inspection covers the text of request and response bodies, including tool call arguments, but does not decode base64 or follow external URLs, so an encoded attachment passes uninspected.
Suppressing the stored payload is a separate lever from blocking the call, and it keeps the metadata.
In practiceFlag or block, in both directions. It does not redact.
Spend limits, and how the watching works
Spend limits track cumulative dollar cost per request against a budget over a rolling or fixed window, scoped by model, provider or custom metadata such as application, team or user. When the budget is reached the gateway blocks further requests until the window resets, with up to 20 rules per gateway. Enforcement is eventually consistent, so a burst of concurrent calls can briefly exceed a threshold before it catches up.
Blocking is one option; the other is to keep serving, by pairing a spend limit on the premium model with a route whose fallback is a cheaper one, so the ceiling degrades quality instead of availability.
What the gateway does not do is warn you: enforcement is a wall, not a signal, and there is no native alerting on spend. Someone has to read the analytics and log APIs against each threshold while there is still room to act. On the estates we run, that reading is ours.
In practiceA wall, not a signal. The alerting is a service, not a feature.
What the prompts carry, once it leaves the gateway, is a data protection question in its own right: the same detection profiles govern email, web uploads and SaaS on preventing data loss across every channel. And if your team builds the applications making these calls, the instrumentation belongs in the build: applications engineered on Cloudflare.
The regulatory hook
What regulators ask about the prompts you send.
When a prompt leaves your infrastructure and reaches a third-party model provider, it crosses a boundary regulators care about. Four questions come back, and three of them are answered by the log.
GDPR and data protection
If a prompt contains personal data, a name, an email address, a customer reference, sending it to a third-party provider is a transfer to a processor. That requires a lawful basis, a data processing agreement and, where the provider sits outside the EEA, a transfer impact assessment. Logging what you send, to whom and when is the evidence base for demonstrating compliance. Purpose limitation and retention rules then apply to those logs too, which is why suppressing the stored payload while keeping the metadata is a useful setting rather than a technicality.
NIS2
For organisations in scope, logging and traceability obligations extend to the systems that support essential and important services, and an LLM application handling customer queries, an internal knowledge base or an operational workflow is one of those systems. The audit trail a gateway produces is a direct input to that documentation: which application called which provider, when, with what policy applied, and what the policy did.
Where your prompts are processed
This is the part most vendors blur, so it is worth stating plainly. A gateway decides which provider receives a call, and proves it in the log. It does not decide where that provider runs the model: inference happens on the provider's infrastructure, under the provider's own regional commitments. Jurisdiction is therefore governed by provider and model selection, backed by contract, and the gateway is what lets you enforce that selection centrally and evidence it afterwards. For a regulated estate, that pairing is the control.
Gulf and GCC sector frameworks
National and sector requirements across the Gulf Cooperation Council apply data localisation and AI governance expectations that are still evolving, generally inside broader operational risk management obligations rather than in a separate regime. Brixio operates within those frameworks from its Dubai hub and maps gateway configuration to the local obligations that apply to your sector.
The wider framework, notification duties included, is on risk management and logging obligations, and the residency half is on data residency and sovereign cloud. None of this is legal advice: engage qualified counsel for your jurisdiction.
Before you sign
What should you check before you call your LLM usage observable?
Five questions to ask of a provider, or of your own teams. None of them needs a tool to answer, and the answers are what a first review starts from.
01
Do you know how many calls leave your applications each day?
If the answer comes from a provider dashboard, the picture is partial. With more than one provider you have several partial pictures that do not add up to a total. Observability means a single count, across every provider, current rather than reconstructed at month end.
02
Do you know what your prompts contain?
Most teams do not, and it is not negligence: prompts are assembled at runtime from user input, database records and application logic, so what ends up in them was never written down anywhere. Personal data, internal references, confidential content. Logging and content inspection are the only reliable way to find out, and the finding is usually a template rather than a person.
03
Are your logs still being written?
The question people expect is how long logs are kept. The question that catches estates out is whether the gateway is still recording at all, because once a storage limit is reached new logs stop being saved. Check the limit, check whether automatic deletion is on, and check that anything you need beyond that limit is being exported.
04
Do you know who pays for which calls?
Provider invoices show total spend. They rarely show which application, team or feature generated it. Without per-application metadata, cost attribution is guesswork, and optimisation is impossible: you cannot reduce a number you cannot break down.
05
What happens if your primary provider goes down?
With no fallback, a provider outage is a service outage. With a fallback hardcoded in application logic, changing it means a deployment, at the worst possible moment. At gateway level it is a configuration change that takes effect immediately, which is the difference between an incident and a routing decision.
Metryx scores your live Cloudflare configuration against a structured baseline, DNS, TLS, WAF and rule posture, which is the ground every gateway decision sits on. It does not inspect your gateway setup, your logging coverage or your spend: run a Cloudflare configuration audit for that baseline, and get the call-volume and prompt-content picture from phase 1 below, since only the gateway itself can produce it.
Not sure what your Cloudflare setup actually covers?
Metryx audits your Cloudflare configuration, DNS, TLS, WAF mode and rule posture, and maps each gap to a fix. It measures the foundation your AI Gateway policies run on; the call baseline and the prompt review run on top of it.
Run an express audit- Free access, no commitment
- Read-only Cloudflare token
- No configuration required
- Downloadable audit report to share internally
- Run as many audits as you want, on as many zones as you want
Your entry point
Where do you begin, depending on where you are?
The right first step depends on what you already hold. Three situations, three different places to begin, and none of them starts with buying a platform.
Whoever ends up owning the watching, the operating model is the one described in managed Cloudflare operations.
01
No idea how many calls leave your applications
Start in observation mode: route calls through a gateway with logging on and no policy enforcement. Application behaviour does not change, and within days you have a count per provider, a cost estimate and a first look at what the prompts carry. That baseline is what every later decision gets argued from.
02
Provider dashboards only, no unified view
Route every provider through one gateway. One endpoint, one audit trail, one cost view, and the only application change is the base URL your SDK points to. Per-application breakdown then comes from metadata rather than from multiplying gateways, which keeps the audit trail single.
03
A control plane in place, but no budgets or alerting
If you already route through a gateway without spend limits, you are observing without acting. Configure thresholds per application and per provider, decide for each one whether it blocks or falls back to a cheaper model, and give the watching an owner, since the gateway enforces but does not warn.
The rollout
How do you instrument LLM calls without slowing the product down?
Latency is the objection that kills these projects, and it deserves a design decision rather than a reassurance. The sequence below adds visibility first, cost control second, and content inspection last, where the trade-off actually lives.
Phase 1
Observation mode
Route calls through the gateway, enable logging, enforce nothing. Cloudflare's network is already in the path for most enterprise estates running ZTNA or WAAP, and with logging alone the addition is a proxy hop. You get the baseline: call volume, provider distribution, token usage, estimated cost. No application code changes.
Phase 2
Per-application attribution
Tag requests with custom metadata: application ID, team, environment. Cost and usage become attributable to what generated them, and the same dimensions later drive spend limits and routing conditions. One gateway, many labels. This is the step that makes cost optimisation actionable, because you cannot optimise what you cannot attribute.
Phase 3
Cache, budgets, then content inspection
Caching goes first: it costs nothing in quality and removes calls entirely. Then spend limits per application and per provider, each with an explicit choice between blocking and falling back. Content inspection goes last, because it is where latency has to be decided rather than promised: scanning requests has no effect on streaming, while scanning responses on a streaming endpoint buffers the whole response before inspecting it, so the first token arrives later. Two documented ways through, picked per endpoint rather than per estate: check requests only where the user is watching tokens appear, or give latency-sensitive traffic its own gateway with its own policy. Policies are set per gateway with no per-request bypass, so that separation is the mechanism rather than a workaround.
Phase 4
Continuous review
Instrumentation decays. New applications appear, providers change pricing, prompt patterns drift, and a threshold set in one quarter stops meaning anything two quarters later. So the last phase is a loop rather than a project: cost against thresholds, DLP events and storage settings, and the rules that turn both into something a human reads. How often depends on how fast your estate moves; what matters is that the loop has an owner.
Observation first is not caution for its own sake. It is what turns the latency conversation into a measurement instead of an argument, before any policy is switched on.
The managed service
How does Brixio operate LLM observability day to day?
Model calls do not keep office hours, and neither does a provider incident. Brixio runs follow-the-sun coverage from four hubs, and an engineer with your gateway in hand takes the response.
24/7 follow-the-sun coverage
- Provider incidents, fallback events, DLP policy hits and spend thresholds watched around the clock
- Coverage is continuous, and the escalation path is the same whatever the hour
- Four hubs, Luxembourg, Paris, Dubai and Singapore
A named, certified team
- Brixio is an Authorized Cloudflare Service Delivery Partner (ASDP), an authorisation held at company level
- The engineers who deliver hold individual Cloudflare certifications, across AI Gateway, Zero Trust and application security
- You have a named point of contact who already holds your gateway configuration, your providers and your compliance constraints
SLA, and a review loop with an owner
- Response SLAs defined per severity, agreed at service start
- The review rhythm is scoped with you, since it depends on how fast your AI estate changes and on what your own governance already asks for
- Thresholds and routing rules revisited whenever an application, a provider or a prompt template changes
Cost and compliance in one place
- Spend read against every threshold while there is still room to act, since the gateway enforces but does not warn
- DLP events, storage limits and export settings reviewed on the same calendar as your other compliance evidence
- The scope we hold for acting on a fallback route without checking back first is written into the engagement rather than assumed
Applications keep calling the providers they already call. The difference is that each call passes one policy plane on the way, and leaves a record behind it.
This runs on Brixio's standing security posture
Cloudflare
Authorized Service Delivery Partner (ASDP)
ISO 27001:2022
certified data handling
450+
delivered projects in regulated industries
4 hubs
Luxembourg · Paris · Dubai · Singapore, follow-the-sun
The umbrella over all of it is AI security with Cloudflare, where the other halves of the subject live. Talk to an expert to scope this against your own environment.
Where it stops
What LLM observability on Cloudflare does not cover.
Four exclusions, stated plainly, because a control plane that implies it covers everything is worse than one with a stated edge. Each of them has a proper home.
Model output quality and evaluation
Whether a response is accurate or relevant is an evaluation question, answered with test sets, benchmarks and human review by the team that owns the use case. A gateway does not score outputs against a reference, run evaluation suites or compare model behaviour across prompt variations, and the AI Gateway feature that once did is deprecated and closed to new accounts. Dedicated tooling is the right answer there, alongside a gateway rather than instead of one. Content safety is a different matter and is in scope, through the guardrail layer, which belongs with prompt protection rather than with metering.
Preventing a provider from training on your data
That is contractual, not technical. A gateway sits between your application and the provider, but once the request is received it does not stop the provider using what it received. Your data processing terms and any opt-out provisions are the control that matters, and reading them is the work. What the gateway gives you is the ability to prove what you sent, and to stop sending what should not go.
Replacing provider billing
Cost figures in a gateway are a best-effort estimation from token counts and model pricing, which the documentation says itself. The provider invoice stays the source of truth, and the two will differ because of negotiated rates, credits and billing cycles. Use gateway data for operational decisions, taken while there is still room to act, and invoices for financial reconciliation.
Unsanctioned AI tools your teams use
AI tools adopted without approval are a network-layer problem, answered by traffic inspection and identity-level policy rather than by an API proxy: your own code never calls them. It is a discovery problem rather than an instrumentation one, and it belongs to shadow AI detection. The two capabilities are complementary, and running one does not give you the other.
Stating the edge is what keeps the rest credible. A control plane that claims to judge model quality or a provider's internals invites a false sense of completeness, and that gets found out. This one answers what left, what it cost, and what it carried.