LLM governance / policy control planePolicy Gateway
Azure APIM AI Gateway vs vendor-agnostic Policy Gateway
Azure APIM AI Gateway focuses on Azure-native policy and monitoring, while Policy Gateway is a vendor-agnostic control plane for multiple model providers.
This comparison highlights when each approach fits and how teams combine them for enterprise governance.
Quick start
Service notes
- Pricing model: Usage-based pricing (~$5 per 1M tokens) billed on total tokens (input + output). See the API pricing page for current plans.
- Data retention: No prompt/output retention by default. Operational telemetry (token counts, timestamps, error codes) is retained for billing and reliability.
- Compatibility: OpenAI-style /v1/chat/completions request and response format with a base URL switch.
- Latency: Depends on model size, prompt length, and load. Streaming reduces time-to-first-token.
- Throughput: Team plans include priority throughput. Actual throughput varies with demand.
- Rate limits: Limits vary by plan and load. Handle 429s with backoff and respect any Retry-After header.
Summary comparison
| Capability | Azure APIM AI Gateway | Vendor-agnostic Policy Gateway |
|---|---|---|
| Provider scope | Azure-centric deployments | Multi-provider (Azure OpenAI, OpenAI, Anthropic, local) |
| LLM policies | API Management AI policy layer | Policy-as-code with rules, reason codes, and rollouts |
| Quotas and chargeback | Usage tracking and rate controls | Per-user and per-project quotas with audit tags |
| Audit exports | Azure-native logging integrations | Splunk, Datadog, Elastic, S3, Azure Monitor |
| Portability | Azure platform alignment | Portable governance across clouds and providers |
Choose Azure APIM AI Gateway when
- Your LLM traffic is entirely inside Azure and you want a single Azure-native control plane.
- You already standardize on Azure API Management policies for API governance.
- Security teams expect Azure Monitor and Log Analytics as the default logging surface.
Choose Policy Gateway when
- You need the same policy across multiple providers or regions.
- Teams want consistent reason codes, audit tags, and rollout controls across apps.
- You want to export audit logs to SIEM tools outside Azure.
They can work together
- Use Azure APIM at the edge for routing and API governance.
- Use Policy Gateway for LLM-specific policy-as-code, quotas, and audit exports.
- Security teams get consistent policy metadata regardless of upstream provider.
Common errors & fixes
- 401 Unauthorized: Check that your API key is set and sent as a Bearer token.
- 404 Not Found: Make sure the base URL ends with /v1 and you call /chat/completions.
- 400 Bad Request: Verify the model id and that messages are an array of { role, content } objects.
- 429 Rate limit: Back off and retry. Use the Retry-After header for pacing.