Skip to main content

[Preview] v1.80.10.rc.1 - Agent Gateway: Azure Foundry & Bedrock AgentCore

Krrish Dholakia
Krrish Dholakia
CEO, LiteLLM
Ishaan Jaff
Ishaan Jaff
CTO, LiteLLM

Deploy this version​

docker run litellm
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.80.10.rc.1

Key Highlights​

Agent Gateway - 4 New Agent Providers​


This release adds support for agents from the following providers:

  • LangGraph Agents - Deploy and manage LangGraph-based agents
  • Azure AI Foundry Agents - Enterprise agent deployments on Azure
  • Bedrock AgentCore - AWS Bedrock agent integration
  • A2A Agents - Agent-to-Agent protocol support

AI Gateway admins can now add agents from any of these providers, and developers can invoke them through a unified interface using the A2A protocol.

For all agent requests running through the AI Gateway, LiteLLM automatically tracks request/response logs, cost, and token usage.

Agent (A2A) Usage UI​

Users can now filter usage statistics by agents, providing the same granular filtering capabilities available for teams, organizations, and customers.

Details:

  • Filter usage analytics, spend logs, and activity metrics by agent ID
  • View breakdowns on a per-agent basis
  • Consistent filtering experience across all usage and analytics views

New Providers and Endpoints​

New Providers (5 new providers)​

ProviderSupported LiteLLM EndpointsDescription
SAP Gen AI Hub/chat/completions, /messages, /responsesSAP Generative AI Hub integration for enterprise AI
LangGraph/chat/completions, /messages, /responses, /a2aLangGraph agents for agentic workflows
Azure AI Foundry Agents/chat/completions, /messages, /responses, /a2aAzure AI Foundry Agents for enterprise agent deployments
Voyage AI Rerank/rerankVoyage AI rerank models support
Fireworks AI Rerank/rerankFireworks AI rerank endpoint support

New LLM API Endpoints (4 new endpoints)​

EndpointMethodDescriptionDocumentation
/containers/{id}/filesGETList files in a containerDocs
/containers/{id}/files/{file_id}GETRetrieve container file metadataDocs
/containers/{id}/files/{file_id}DELETEDelete a file from a containerDocs
/containers/{id}/files/{file_id}/contentGETRetrieve container file contentDocs

New Models / Updated Models​

New Model Support (270+ new models)​

ProviderModelContext WindowInput ($/1M tokens)Output ($/1M tokens)Features
OpenAIgpt-5.2400K$1.75$14.00Reasoning, vision, PDF, caching
OpenAIgpt-5.2-pro400K$21.00$168.00Reasoning, web search, vision
Azureazure/gpt-5.2400K$1.75$14.00Reasoning, vision, PDF, caching
Azureazure/gpt-5.2-pro400K$21.00$168.00Reasoning, web search
Bedrockus.writer.palmyra-x4-v1:0128K$2.50$10.00Function calling, PDF input
Bedrockus.writer.palmyra-x5-v1:01M$0.60$6.00Function calling, PDF input
Bedrockeu.anthropic.claude-opus-4-5-20251101-v1:0200K$5.00$25.00Reasoning, computer use, vision
Bedrockgoogle.gemma-3-12b-it128K$0.10$0.30Audio input
Bedrockmoonshot.kimi-k2-thinking128K$0.60$2.50Reasoning
Bedrocknvidia.nemotron-nano-12b-v2128K$0.20$0.60Vision
Bedrockqwen.qwen3-next-80b-a3b128K$0.15$1.20Function calling
Vertex AIvertex_ai/deepseek-ai/deepseek-v3.2-maas164K$0.56$1.68Reasoning, caching
Mistralmistral/codestral-2508256K$0.30$0.90Function calling
Mistralmistral/devstral-2512256K$0.40$2.00Function calling
Mistralmistral/labs-devstral-small-2512256K$0.10$0.30Function calling
Cerebrascerebras/zai-glm-4.6128K--Chat completions
NVIDIA NIMnvidia_nim/ranking/nvidia/llama-3.2-nv-rerankqa-1b-v2-FreeFreeRerank
Voyagevoyage/rerank-2.532K$0.05/1K tokens-Rerank
Fireworks AI227 new modelsVariousVariousVariousFull model catalog

Features​

Bug Fixes​

LLM API Endpoints​

Features​

Bugs​

  • General
    • Fix handle string content in is_cached_message - PR #17853

Management Endpoints / UI​

Features​

  • UI Settings
    • Add Get and Update Backend Routes for UI Settings - PR #17689
    • UI Settings page implementation - PR #17697
    • Ensure Model Page honors UI Settings - PR #17804
    • Add All Proxy Models to Default User Settings - PR #17902
  • Agent & Usage UI
  • Logs & Spend
    • Enhanced spend analytics in logs view - PR #17623
    • Add user info delete modal for user management - PR #17625
    • Show request and response details in logs view - PR #17928
  • Virtual Keys
    • Fix x-litellm-key-spend header update - PR #17864
  • Models & Endpoints
    • Model Hub Useful Links Rearrange - PR #17859
    • Create Team Model Dropdown honors Organization's Models - PR #17834
  • SSO & Auth
    • Allow upserting user role when SSO provider role changes - PR #17754
    • Allow fetching role from generic SSO provider (Keycloak) - PR #17787
    • JWT Auth - allow selecting team_id from request header - PR #17884
    • Remove SSO Config Values from Config Table on SSO Update - PR #17668
  • Teams
    • Attach team to org table - PR #17832
    • Expose the team alias when authenticating - PR #17725
  • MCP Server Management
    • Add extra_headers and allowed_tools to UpdateMCPServerRequest - PR #17940
  • Notifications
    • Show progress and pause on hover for Notifications - PR #17942
  • General
    • Allow Root Path to Redirect when Docs not on Root Path - PR #16843
    • Show UI version number on top left near logo - PR #17891
    • Re-organize left navigation with correct categories and agents on root - PR #17890
    • UI Playground - allow custom model names in model selector dropdown - PR #17892

Bugs​

  • UI Fixes
    • Fix links + old login page deprecation message - PR #17624
    • Filtering for Chat UI Endpoint Selector - PR #17567
    • Race Condition Handling in SCIM v2 - PR #17513
    • Make /litellm_model_cost_map public - PR #16795
    • Custom Callback on UI - PR #17522
    • Add User Writable Directory to Non Root Docker for Logo - PR #17180
    • Swap URL Input and Display Name inputs - PR #17682
    • Change deprecation banner to only show on /sso/key/generate - PR #17681
    • Change credential encryption to only affect db credentials - PR #17741
  • Auth & Routes
    • Return 403 instead of 503 for unauthorized routes - PR #17723
    • AI Gateway Auth - allow using wildcard patterns for public routes - PR #17686

AI Integrations​

New Integrations (4 new integrations)​

IntegrationTypeDescription
SumoLogicLoggingNative webhook integration for SumoLogic - PR #17630
Arize PhoenixPrompt ManagementArize Phoenix OSS prompt management integration - PR #17750
SendgridEmailSendgrid email notifications integration - PR #17775
OnyxGuardrailsOnyx guardrail hooks integration - PR #16591

Logging​

  • Langfuse
    • Propagate Langfuse trace_id - PR #17669
    • Prefer standard trace id for Langfuse logging - PR #17791
    • Move query params to create_pass_through_route call in Langfuse passthrough - PR #17660
    • Add support for custom masking function - PR #17826
  • Prometheus
    • Add 'exception_status' to prometheus logger - PR #17847
  • OpenTelemetry
    • Add latency metrics (TTFT, TPOT, Total Generation Time) to OTEL payload - PR #17888
  • General
    • Add polling via cache feature for async logging - PR #16862

Guardrails​

Prompt Management​

  • General
    • New API for integrating prompt management providers - PR #17829

Spend Tracking, Budgets and Rate Limiting​

  • Service Tier Pricing - Extract service_tier from response/usage for OpenAI flex pricing - PR #17748
  • Agent Cost Tracking - Track agent_id in SpendLogs - PR #17795
  • Tag Activity - Deduplicate /tag/daily/activity metadata - PR #16764
  • Rate Limiting - Dynamic Rate Limiter - allow specifying ttl for in memory cache - PR #17679

MCP Gateway​

  • Chat Completions Integration - Add support for using MCPs on /chat/completions - PR #17747
  • UI Session Permissions - Fix UI session MCP permissions across real teams - PR #17620
  • OAuth Callback - Fix MCP OAuth callback routing and URL handling - PR #17789
  • Tool Name Prefix - Fix MCP tool name prefix - PR #17908

Agent Gateway (A2A)​

  • Cost Per Query - Add cost per query for agent invocations - PR #17774
  • Token Counting - Add token counting non streaming + streaming - PR #17779
  • Cost Per Token - Add cost per token pricing for A2A - PR #17780
  • LangGraph Provider - Add LangGraph provider for Agent Gateway - PR #17783
  • Bedrock & LangGraph Agents - Allow using Bedrock AgentCore, LangGraph agents with A2A Gateway - PR #17786
  • Agent Management - Allow adding LangGraph, Bedrock Agent Core agents - PR #17802
  • Azure Foundry Agents - Add Azure AI Foundry Agents support - PR #17845
  • Azure Foundry UI - Allow adding Azure Foundry Agents on UI - PR #17909
  • Azure Foundry Fixes - Ensure Azure Foundry agents work correctly - PR #17943

Performance / Loadbalancing / Reliability improvements​

  • Memory Leak Fix - Cut memory leak in half - PR #17784
  • Spend Logs Memory - Reduce memory accumulation of spend_logs - PR #17742
  • Router Optimization - Replace time.perf_counter() with time.time() - PR #17881
  • Filter Internal Params - Filter internal params in fallback code - PR #17941
  • Gunicorn Suggestion - Suggest Gunicorn instead of uvicorn when using max_requests_before_restart - PR #17788
  • Pydantic Warnings - Mitigate PydanticDeprecatedSince20 warnings - PR #17657
  • Python 3.14 Support - Add Python 3.14 support via grpcio version constraints - PR #17666
  • OpenAI Package - Bump openai package to 2.9.0 - PR #17818

Documentation Updates​

  • Contributing - Update clone instructions to recommend forking first - PR #17637
  • Getting Started - Improve Getting Started page and SDK documentation structure - PR #17614
  • JSON Mode - Make it clearer how to get Pydantic model output - PR #17671
  • drop_params - Update litellm docs for drop_params - PR #17658
  • Environment Variables - Document missing environment variables and fix incorrect types - PR #17649
  • SumoLogic - Add SumoLogic integration documentation - PR #17647
  • SAP Gen AI - Add SAP Gen AI provider documentation - PR #17667
  • Authentication - Add Note for Authentication - PR #17733
  • Known Issues - Adding known issues to 1.80.5-stable docs - PR #17738
  • Supported Endpoints - Fix Supported Endpoints page - PR #17710
  • Token Count - Document token count endpoint - PR #17772
  • Overview - Made litellm proxy and SDK difference cleaner in overview with a table - PR #17790
  • Containers API - Add docs for containers files API + code interpreter on LiteLLM - PR #17749
  • Target Storage - Add documentation for target storage - PR #17882
  • Agent Usage - Agent Usage documentation - PR #17931, PR #17932, PR #17934
  • Cursor Integration - Cursor Integration documentation - PR #17855, PR #17939
  • A2A Cost Tracking - A2A cost tracking docs - PR #17913
  • Azure Search - Update azure search docs - PR #17726
  • Milvus Client - Fix milvus client docs - PR #17736
  • Streaming Logging - Remove streaming logging doc - PR #17739
  • Integration Docs - Update integration docs location - PR #17644
  • Links - Updated docs links for mistral and anthropic - PR #17852
  • Community - Add community doc link - PR #17734
  • Pricing - Update pricing for global.anthropic.claude-haiku-4-5-20251001-v1:0 - PR #17703
  • gpt-image-1-mini - Correct model type for gpt-image-1-mini - PR #17635

Infrastructure / Deployment​

  • Docker - Use python instead of wget for healthcheck in docker-compose.yml - PR #17646
  • Helm Chart - Add extraResources support for Helm chart deployments - PR #17627
  • Helm Versioning - Add semver prerelease suffix to helm chart versions - PR #17678
  • Database Schema - Add storage_backend and storage_url columns to schema.prisma for target storage feature - PR #17936

New Contributors​

  • @xianzongxie-stripe made their first contribution in PR #16862
  • @krisxia0506 made their first contribution in PR #17637
  • @chetanchoudhary-sumo made their first contribution in PR #17630
  • @kevinmarx made their first contribution in PR #17632
  • @expruc made their first contribution in PR #17627
  • @rcII made their first contribution in PR #17626
  • @tamirkiviti13 made their first contribution in PR #16591
  • @Eric84626 made their first contribution in PR #17629
  • @vasilisazayka made their first contribution in PR #16053
  • @juliettech13 made their first contribution in PR #17663
  • @jason-nance made their first contribution in PR #17660
  • @yisding made their first contribution in PR #17671
  • @emilsvennesson made their first contribution in PR #17656
  • @kumekay made their first contribution in PR #17646
  • @chenzhaofei01 made their first contribution in PR #17584
  • @shivamrawat1 made their first contribution in PR #17733
  • @ephrimstanley made their first contribution in PR #17723
  • @hwittenborn made their first contribution in PR #17743
  • @peterkc made their first contribution in PR #17727
  • @saisurya237 made their first contribution in PR #17725
  • @Ashton-Sidhu made their first contribution in PR #17728
  • @CyrusTC made their first contribution in PR #17810
  • @jichmi made their first contribution in PR #17703
  • @ryan-crabbe made their first contribution in PR #17852
  • @nlineback made their first contribution in PR #17851
  • @butnarurazvan made their first contribution in PR #17468
  • @yoshi-p27 made their first contribution in PR #17915

Full Changelog​

View complete changelog on GitHub