Fallbacks

Automatically switch to backup LLMs when the primary fails.

Available on all Portkey plans.

Specify a prioritized list of providers/models. If the primary LLM fails, Portkey automatically falls back to the next in line.

Examples

Between Models

{
  "strategy": { "mode": "fallback" },
  "targets": [
    { "override_params": { "model": "@openai-prod/gpt-4o" } },
    { "override_params": { "model": "@anthropic-prod/claude-3-5-sonnet-20241022" } }
  ]
}

The @provider-slug/model-name format automatically routes to the correct provider. Set up providers in Model Catalog.

Create and use configs in your requests.

Trigger on Specific Status Codes

By default, fallback triggers on any non-2xx status code.

Customize with on_status_codes:

{
  "strategy": { "mode": "fallback", "on_status_codes": [429, 503] },
  "targets": [
    { "provider": "@openai-prod" },
    { "provider": "@azure-prod" }
  ]
}

Tracing Fallback Requests

Portkey logs all requests in a fallback chain. To trace:

1. Filter logs by Config ID to see all requests using that config
2. Filter by Trace ID to see all attempts for a single request

Considerations

• Ensure fallback LLMs are compatible with your use case
• A single request may invoke multiple LLMs
• Each LLM has different latency and pricing